Reexamining Learning Curve Analysis in Programming Education: The Value of Many Small Problems

[to appear in EDM 2024] Analyzing which skills students develop in introductory programming education is an important question for the computer science education community. Knowledge components, units of knowledge that can be measured by performance on a set of tasks, can be used to identify key concepts and skills in a given domain. While knowledge components in other domains have been successfully identified using learning curve analysis, such attempts on students' open-ended code-writing assignments have not been very successful. To understand why, we replicated a previously proposed approach, which uses abstract syntax tree (AST) nodes as knowledge components, on data collected across multiple semesters of a large-scale introductory programming course. Findings from our replication show that, given sufficient measurement opportunities, AST nodes can provide a viable knowledge component model for learning curve analysis to understand student learning, contrary to earlier findings. In addition to providing evidence for the validity of AST-based knowledge components, we recommend a set of conditions for programming courses that enable knowledge components generated using AST nodes to be successfully observed using learning curve analysis. This supports the integration of learning curves as a tool in any environment collecting code-writing data to be used for modifying the curriculum based on student performance and monitoring the development of skills related to language elements.