1001 Games a Night—Continuous Evaluation of an Intelligent Multi-agent-Based System

The goal of the presented approach is to improve the stability of our RoboCup team code by providing an improved continuous integration software engineering process. As big and even small changes in our code base cannot be judged by just a couple of games, roughly 1000 games were run each night to have a good feeling whether changes were for the better or for worse. In addition, it is now possible to analyze the output to gain even deeper understanding of different approaches and parameters. This is supported by interactive visualization techniques. As a brute force approach will collect way too much data every night, it is necessary to condense the output and keep just a very small fraction of the detailed log data for further analysis. To decide which log files to keep different outlier detection algorithms are compared and optimized.