Analysis of Learning Behavior in an Automated Programming Assessment Environment: A Code Quality Perspective

Abstract

Automated programming assessment systems are useful tools to track the learning progress of students automatically and thereby reduce the workload of educators. They can also be used to gain insights into how students learn, making it easier to formulate strategies aimed at enhancing learning performance. Rather than functional code which is always inspected, code quality remains an essential aspect to which not many educators consider when designing an automated programming assessment system. In this study, we applied data mining techniques to analyze the results of an automated assessment system to reveal unexpressed patterns in code quality improvement that are predictive of final achievements in the course. Cluster analysis is first utilized to categorize students according to their learning behavior and outcomes. Cluster profile analysis is then leveraged to highlight actionable factors that could affect their final grades. Finally, the same factors are employed to construct a classification model by which to make early predictions of the students' final results. Our empirical results demonstrate the efficacy of the proposed scheme in providing valuable insights into the learning behaviors of students in novice programming courses, especially in code quality assurance, which could be used to enhance programming performance at the university level. INDEX TERMS automated programming assessment system, code quality, educational data mining, early learning achievement detection, programming education

Putting It All Together: Combining Learning Analytics Methods and Data Sources to Understand Students’ Approaches to Learning Programming

“…Reporting distinct groups of programming learners is common in the learning analytics literature; however, there are significant differences among disciplines and cluster characteristics pertaining to the context [12,20,31,35,56]. By clustering students' learning tactics, [12] found three distinct groups of learners (highly selective, strategic, intensive), with very similar profiles and performance to those found in this work.…”

“…Blikstein [18] used the data captured from a programming environment to analyze the changes in the number of code compilations over time, identify moments of greater difficulty in the programming process, and trace an approximation of each prototypical coding profile and style. In the work by Chen et al [20], cluster analysis confirmed that learning motivation (denoted by early and on-time submissions) was strongly associated with final grades, and not the number of submissions made by students. Furthermore, Watson et al [34] developed a unique approach for predicting performance based upon how a student responds to different types of errors compared to their peers.…”

“…Accounting for the individual time-related characteristics has allowed to illustrate how events occur within the flow of continuous events in a particular time window. However, the aforementioned studies analyzed data exclusively from programming environments [18,20,[33][34][35] or from LMSs alone [12]. Learning programming involves both understanding theoretical concepts and acquiring practical skills.…”

“…By clustering students' learning tactics, [12] found three distinct groups of learners (highly selective, strategic, intensive), with very similar profiles and performance to those found in this work. Conversely, [20] identified five distinct clusters of students according to the overall frequency of their learning actions (effective, high-effort, average, blind-trial, low-effort) and found that early and on-time submissions were more highly associated with final grades than the number of assignment submissions. Lastly, [35] identified four different student profiles according to their programming patterns (quitters, approachers, knowers and solvers), each presenting different performance levels.…”

“…With this in mind, authors have analyzed the data collected from auxiliary systems that support programming students in their learning process (e.g. automated assessment systems for programming assignments), to examine how different factors affect students' outcomes and programming mastery, such as code quality, code size, and usage frequency [18][19][20]. In this study, we argue that a holistic approach that is based on the analysis of data originating from several relevant sources to programming education would provide a more accurate view of students' learning, considering how the temporality of the learning tactics, the interconnectedness of events, and the combination of various learning strategies may influence the outcome of the learning process [21].…”