Predicting and Explaining Automatic Testing Tool Effectiveness

Automatic white-box test generation is a challenging problem. Many existing tools rely on complex code analyses and heuristics. As a result, structural features of an input program may impact tool effectiveness in ways that tool users and designers may not expect or understand. We develop a technique that uses structural pro- gram metrics to both predict and explain the test cover- age achieved by three automatic test generation tools. We use coverage and structural metrics extracted from 11 software projects to train several decision-tree clas- sifiers. These classifiers can predict high or low cover- age with success rates of 82% to 94%. In addition, they show tool users and designers the program structures that impact tool effectiveness.