A Comparison of Open-Source Static Analysis Tools for Vulnerability Detection in C/C++ Code

We describe work that is part of a research project on static code analysis between the Alexandru Ioan Cuza University and Bitdefender. The goal of the project is to develop customized static analysis tools for detecting potential vulnerabilities in C/C++ code. We present the results of benchmarking several existing open source static analysis tools for C/C++ against the Toyota ITC test suite [1] in order to determine which tools are best suited to our purpose. The Toyota ITC test suite is a synthetic benchmark for C/C++ consisting of around 650 test cases organized by defect type and defect subtype and is well-suited to our purpose, since it contains various bugs such as buffer overflows that are common in C/C++ code. We analyze the open-source static analysis tools according to the existing quality indicators such as detection rate and false positive rate proposed in [1], but we also introduce a new quality metric that we call robust detection which also allows us to measure unique detections by tool and by (sub)defect type. We also find several mistakes in the Toyota ITC testsuite that we fix. We publish the harness used to benchmark the static analyzers in order for anyone to be able to reproduce our results.