Smart generation of code tracing questions for assessment in introductory programming

Abstract

Teaching programming is a challenging activity nowadays, especially in introductory programming courses, which are typically massively attended. Writing functional programs is a cognitive skill, which many students, novices in programming, find it difficult to master. It is equally challenging to assess this ability of students. Research has shown that students need to learn how to read and understand, before they can learn to write programs. Code tracing questions are a suitable way to assess the knowledge of semantics of programming constructs (understanding what programs do). However, when designing these types of questions, teachers need to be aware of the complexity of the code and must always try to create questions with code snippets of same or similar complexity, especially when preparing different versions of the same test (for large student groups). In this paper we propose a new model for smart automatic generation of code tracing questions. The model uses a suitable source code metric to measure the code complexity, and to enable generation of questions containing code snippets of consistent complexity. We also introduce CodeCPP: a newly developed Moodle plugin, which is an implementation of the proposed model. The evaluation of CodeCPP, through a year‐long use of the system in teaching programming as well as feedback from selected teachers with many years of experience in teaching programming courses, shows positive and promising results. The system utilization confirms that the application of the proposed model enables a fair and objective knowledge assessment. Furthermore, the feedback provided by teachers and students is highly positive.

Automated generation and correction of diagram‐based exercises for Moodle

“…Also, within programming, another possibility is asking tracing questions about the execution of programs. Some approaches can generate these questions automatically [31] using a complexity metric to select code snippets of similar complexity. As we will see in Section 5.2, Wodel‐Edu can automatically generate diagrams (i.e., solutions matching an exercise statement) of similar complexity by using size parameters and constraints.…”

Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses

“…Manual question creation by instructors (Sekiya and Yamaguchi, 2013;Hassan and Zilles, 2021) is time-consuming and lacks scalability. Automatic generation using program analysis saves time, yet is limited by the analyzer's capabilities and lacks question diversity (Zavala and Mendoza, 2018;Thomas et al, 2019;Russell, 2021;Lehtinen et al, 2021;Stankov et al, 2023).…”