Helping Student Programmers Through Industrial-Strength Static Analysis: A Replication Study

ABSTRACTStatic analysis tools evaluate source code to identify problems beyond typical compiler errors. Prior work has shown a statistically significant relationship between correctness and static analysis results. This paper replicates and extends a prior study on FindBugs, a static analysis tool aimed at professional Java programmers. The prior study showed a strong link between certain FindBugs issues and problems with program correctness. It also showed they were significantly associated with struggling, as indicated by taking more time, making more submissions, and receiving lower scores. However, the study used small programming exercises involving only a handful of lines of code from one semester of a CS1 course. This replication study uses the same experimental approach, but applies it to full-scale programming assignments from hundreds to thousands of lines in length, across all sections of CS1, CS2, and CS3 at a large public university over a period of 4 academic semesters, and involving 4,244 students in 109 laboratory sections completing 255,222 submission attempts. The goal of this replication study is to confirm how prior results hold up, and to explore how the results apply to full-sized programming assignments. We find a set of FindBugs warnings that are inversely correlated with correctness and confirm that their presence is still significantly associated with struggling on larger programming assignments. However, a larger number of FindBugs issues were identified as valuable. We also discuss how student-friendly messages have been added to provided student-readable feedback for a tool where the native messages were written for professional programmers.