Improving marking efficiency for longer programming solutions based on a semi‐automated assessment approach

Abstract

In recent years, many students in higher education have begun to learn programming languages. In doing so they will complete a variety of programming tasks of varying degrees of complexity. The students need to get consistent and personalized feedback to develop their programming skills. Human markers can provide personalized feedback using traditional manual approaches to assessment, but they may provide inconsistent feedback (especially for long programming solutions) since marking the programming solutions of multiple students can represent a significant workload for them. While full‐automated assessment systems are the best to provide consistent feedback, they may not provide sufficiently personalized feedback for novice programmers. This study develops a novel semi‐automated assessment approach in order to improve efficiency of human marker in the marking process and increase consistency of feedback (for both short and long programming solutions). It advocates the reuse of human marker's comments for similar code snippets, defined as segmented marking in this study. New full and partial marking models are developed based on segmented marking and they are tested by expert markers. The findings show that the two models are similar in efficiency, but that a partial marking approach potentially offers an improved efficiency for longer programming solutions. Such a finding has significant potential to reduce time spent on marking throughout the sector, which would have significant impact on both resourcing and timeliness of feedback.

Tools and Techniques for Teaching Computer Programming: A Review

“…Some instructors recommend semiautomated systems (Buyrukoglu et al., 2019) or fully automated systems (Restrepo‐Calle et al., 2019; Ullah et al., 2018) to evaluate programs written by students.…”

Semiautomatic generation and assessment of Java exercises in engineering education

“…As MOOCs have become a popular mode of learning in the last ten years, the study of automatic assessment (AA) in educational contexts is a research field that has been deeply studied lately [11,17,31,45,47,49,52,8,9]. The development of an AA system is often based on different key factors [30,38].…”

“…Output comparison techniques also present another important limitation: because they cannot access the source code and they only check whether the output is correct or wrong, they have only two possible evaluation values: correct or wrong. Nevertheless, the real scenario is often different: teachers usually divide the mark of an exercise into smaller evaluable parts [15,26,9]. Assuming a methodology where 10 points is the maximum grade, 5 the minimum grade to pass the exam, and each appraisable piece not fulfilled decreases the student's final mark, Solution 1 would barely pass the exam.…”

“…These systems consider different aspects of the code that can be classified as dynamic , related to the execution of the code (e.g., the comparison of the output values obtained for both the teacher's and student's solution with a certain set of inputs); or static , related to the structure of the code (e.g., the correct use of reusability resources or the coding style) [43]. Traditionally, dynamic aspects are associated with automated assessment while static aspects are considered to be assessed manually [9].…”

“…This system's performance increases with the number of corrected exercises in the pool, so it is quite useful in MOOCs, where there is a large number of exercise submissions. Another recent AA system of this kind was proposed by Buyrukoglu et al [9]. In their approach, according to a similarity threshold, similar codes are grouped.…”