Automated Grading and Feedback Tools for Programming Education: A Systematic Review

Abstract

We conducted a systematic literature review on automated grading and feedback tools for programming education. We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation and evaluation techniques. Most papers assess the correctness of assignments in object-oriented languages. Typically, these tools use a dynamic technique, primarily unit testing, to provide grades and feedback to the students or static analysis techniques to compare a submission with a reference solution or with a set of correct student submissions. However, these techniques’ feedback is often limited to whether the unit tests have passed or failed, the expected and actual output, or how they differ from the reference solution. Furthermore, few tools assess the maintainability, readability or documentation of the source code, with most using static analysis techniques, such as code quality metrics, in conjunction with grading correctness. Additionally, we found that most tools offered fully automated assessment to allow for near-instantaneous feedback and multiple resubmissions, which can increase student satisfaction and provide them with more opportunities to succeed. In terms of techniques used to evaluate the tools’ performance, most papers primarily use student surveys or compare the automatic assessment tools to grades or feedback provided by human graders. However, because the evaluation dataset is frequently unavailable, it is more difficult to reproduce results and compare tools to a collection of common assignments.

Assessment Automation of Complex Student Programming Assignments

“…The biggest problem with this study was that the existing pre-built tools did not have all the elements we needed, so a semi-automated option was the best approach. As stated in [8] "They typically struggle, however, to grade design quality aspects and to identify student misunderstandings". For an in-depth comparison of pre-built tools, we recommend looking at Table 1 in [3] where 30 tools were compared.…”

“…A lot of papers [13][14][15] have focused on introductory programming courses, but there is an open space in the domain of more complex assignments. Also, researchers often focus on building different tools (as seen from [8]) rather than focusing on the process that goes into it. This paper fills this gap.…”

“…Beyond the tools mentioned in such surveys, there are other articles where a new tool was developed. The most recent systematic review on the topic "Automated Grading and Feedback Tools for Programming Education" can be found in [8].…”

Exploring the Impact of Grading for Equity on Student’s Academic Performance: A Qualitative Study

“…In an inclusive approach to teaching, grading requires focusing on justice, individualised comments, and assistance with targeted support (Messer et al, 2024). As an educator whose goal is to promote a learning process that contradicts traditional grading practices, we often tell our students that mistake is a process of learning or even desired for successful learning yet, we include some traditional ways of assessing performances or outputs.…”