Back to the Future! Studying Data Cleanness in Defects4J and its Impact on Fault Localization
CoRR(2023)
摘要
For software testing research, Defects4J stands out as the primary benchmark
dataset, offering a controlled environment to study real bugs from prominent
open-source systems. However, prior research indicates that Defects4J might
include tests added post-bug report, embedding developer knowledge and
affecting fault localization efficacy. In this paper, we examine Defects4J's
fault-triggering tests, emphasizing the implications of developer knowledge of
SBFL techniques. We study the timelines of changes made to these tests
concerning bug report creation. Then, we study the effectiveness of SBFL
techniques without developer knowledge in the tests. We found that 1) 55% of
the fault-triggering tests were newly added to replicate the bug or to test for
regression; 2) 22% of the fault-triggering tests were modified after the bug
reports were created, containing developer knowledge of the bug; 3) developers
often modify the tests to include new assertions or change the test code to
reflect the changes in the source code; and 4) the performance of SBFL
techniques degrades significantly (up to --415% for Mean First Rank) when
evaluated on the bugs without developer knowledge. We provide a dataset of bugs
without developer insights, aiding future SBFL evaluations in Defects4J and
informing considerations for future bug benchmarks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要