The Tip of the Iceberg: On the Merits of Finding Security Bugs

AbstractIn this article, we investigate a fundamental question regarding software security: Is the security of SW releases increasing over time? We approach this question with a detailed analysis of the large body of open-source software packaged in the popular Debian GNU/Linux distribution. Contrary to common intuition, we find no clear evidence that the vulnerability rate of widely used software decreases over time: Even in popular and “stable” releases, the fixing of bugs does not seem to reduce the rate of newly identified vulnerabilities. The intuitive conclusion is worrisome: Commonly employed development and validation procedures do not seem to scale with the increase of features and complexity—they are only chopping pieces off the top of an iceberg of vulnerabilities.To the best of our knowledge, this is the first investigation into the problem that studies a complete distribution of software, spanning multiple versions. Although we can not give a definitive answer, we show that several popular beliefs also cannot be confirmed given our dataset. We publish our Debian Vulnerability Analysis Framework (DVAF), an automated dataset creation and analysis process, to enable reproduction and further analysis of our results. Overall, we hope our contributions provide important insights into the vulnerability discovery process and help in identifying effective techniques for vulnerability analysis and prevention.