Experiences of Using Automated Assessment in Computer Science Courses

Abstract

In this paper we discuss the use of automated assessment in a variety of computer science courses that have been taught at Israel Academic College by the authors. The course assignments were assessed entirely automatically using Checkpoint, a web-based automated assessment framework. The assignments all used free-text questions (where the students type in their own answers). Students were allowed to correct errors based on feedback provided by the system and resubmit their answers. A total of 141 students were surveyed to assess their opinions of this approach, and we analysed their responses. Analysis of the questionnaire showed a low correlation between questions, indicating the statistical independence of the individual questions. As a whole, student feedback on using Checkpoint was very positive, emphasizing the benefits of multiple attempts, impartial marking, and a quick turnaround time for submissions. Many students said that Checkpoint gave them confidence in learning and motivation to practise. Students also said that the detailed feedback that Checkpoint generated when their programs failed helped them understand their mistakes and how to correct them.

Combining Summative and Formative Evaluation Using Automated Assessment

“…Students have expressed satisfaction with automated assessment in the past (Rosenthal, Suppes, & Ben-Zvi, 2013), feeling that the instant feedback that it provides, as well as consistency of marking, is an advantage over manual marking. The results of our earlier survey of the courses discussed in this paper (English & English 2015) certainly reinforce that view. Student feedback on using Checkpoint was very positive, emphasizing the benefits of multiple attempts and a quick turnaround time for submissions, as well as the ability to work at their own pace and in their own time.…”

“…A detailed empirical analysis of student reactions to Checkpoint was already published in an earlier paper (English & English, 2015) to which interested readers should refer. It described the results of a survey of 141 students on four of the computer science courses described above.…”

Investigating the Feasibility of Automatic Assessment of Programming Tasks

“…Many benefits of applying automatic assessment of programming assignments have been reported. Automatic assessment is more likely to be consistent and objective (Arifi, Abdellah, Zahi, & Benabbou, 2015;Staubitz, Klement, Teusner, Renz, & Meinel, 2016), enables rapid feedback (Arifi et al, 2015;Liu et al, 2016;Nordquist, 2007;Poon et al, 2018), and allows for students to submit multiple improved versions of the programs they have written (Del Fatto et al, 2017;English & English, 2015;Malmi, Korhonen, & Saikkonen, 2002;Pettit, Homer, Gee, Mengel, & Starbuck, 2015;Staubitz et al, 2016). Automatic assessment can play a motivational role in engaging students in the educational process (Šťastná, Juhár, Biňas, & Tomášek, 2015;Staubitz et al, 2016).…”

“…Staubitz et al (2016) describe a number of challenges associated with applying automatic assessment of programming tasks. An important challenge, often overlooked, is that considerable time and effort need to be devoted to the implementation of resources for automated assessment (Ala-Mutka, 2005; (English & English, 2015;Pieterse, 2013). Another problem is that the development of new exercises often requires considerable technical skills beyond the scope of the content being assessed (Korhonen & Malmi, 2000;Pieterse, 2013).…”

Lecturers’ and Students’ Experiences with an Automated Programming Assessment System

“…Automated programming assessment systems (APAS) are used for overcoming problems associated with manually managed programming assignments, such as objective and efficient assessments in large classes and providing timely, targeted, and helpful feedback (Mekterovic and Brkic, 2017). During the last years, a large variety of APAS appeared on the market (Keuning et al, 2016(Keuning et al, , 2018Mekterović et al, 2020), like Checkpoint (English and English, 2015), JACK (Goedicke et al, 2008) or ArTEMiS (Krusche and Seitz, 2018). The main objectives of these APAS are to motivate students, provide a well-founded overview of the learning progress, improve the quality of teaching and students' contributions, minimize the programming entry hurdle, standardize and objectify the feedback and decrease dropout rates (Keuning et al, 2018;Mekterović et al, 2020).…”