LEGenT: Localizing Errors and Generating Testcases for CS1

In a CS1 course, testcases are the most common way of providing feedback. However, manually designed testcases, even if carefully crafted, may miss out on certain crucial corner cases. These testcases are only generated once for the whole class and do not take into account errors generated by specific students. This paper presents LEGenT, an automated tool that generates personalized testcases for student submission. LEGenT first localizes a statement in the program that causes deviation from the expected behaviour. Then it generates testcases that expose the deviation to the student. Our premise is that such a targeted test would help students identify one of the early reasons for the deviation. LEGenT works by separating buggy programs from the correct ones using an off-the-shelf formal equivalence checker. It uses another off-the-shelf tool to cluster buggy as well as correct programs. It aligns an incorrect program cluster with a nearby correct program cluster to generate relevant testcases for the incorrect program. We have evaluated our technique on 26082 (2764 correct + 23318 buggy) real student submissions across 11 different programming problems. LEGenT was able to localize an erroneous statement and generated testcases for 7696 buggy submissions. LEGenT fails to generate a testcase when the buggy submission is nowhere close to any correct submission. For the cases where LEGenT successfully generated testcases, we use a novel validation method to prove its accuracy, that LEGenT correctly identifies a cause of deviation from the correct behaviour.