Are Automated Assessment Tools Helpful in Programming Courses?

Abstract

Raymond S. Pettit teaches courses in programming, artificial intelligence, objected oriented design, algorithms, theory of computation, and related subjects in ACU's School of Information Technology and Computing. Prior to joining the ACU faculty, he spent twenty years in software development, research, and training the Air Force Research Lab and NASA's Langley Research Center as well as private industry. His current research focuses on how automated assessment tools interact with student learning in university programming courses. Are Automated Assessment Tools Helpful in ProgrammingCourses? AbstractAutomated assessment tools (AATs) are growing in popularity in introductory programming courses, but researchers may have a difficult time synthesizing valid data to draw conclusions about the tools' usefulness. Our first step addressing this issue was to break down our overriding question-are automated assessment tools helpful in programming courses?-into four more specific questions: (1) Have AATs proven to be helpful in improving student learning? (2) Do students think that AATs have improved their performance? (3) After having used the tools, do instructors think that the tools have improved their teaching experiences? and (4) Is the assessment performed by AATs accurate enough to be helpful? In discussing the many AATs that exist, many researchers have only reported results relevant to one or two of these specific questions. We address each of our four questions separately and draw on data from 24 different tools to arrive at our conclusions. We determine that the literature demonstrates AATs helpfulness in student learning, instructor support, and assessment accuracy. However, we found results about students' opinions regarding the helpfulness of AATs to be inconclusive. Given our findings, we make suggestions both for instructors using these tools and to researchers creating them.

The effect of automatic assessment on novice programming: Strengths and limitations of existing systems

“…According to Douce et al [9], an automated assignment testing system was perhaps the earliest system developed by Hollingsworth [19] in which students submit their assemblylanguage-written program using punch cards for AA. In the literature, researchers have discussed numerous automated assessment tools in their articles [34], for example, eGrader [45], CourseMarker [18], GAME [6], ASSYST [22], PASS [49], QUIMERA [14], Web-Cat [11,12], BOSS [23], TRY [38], AutoLEP [53], PETCHA [35], Automata [46], EduPCR [54], Scheme-robo [43], JavaMarker [1], Kassandra [52], SAC [4], Ceilidh [5], Athene [50], Mooshak [27], Auto-GradeMe [61], SSBG [55], and much more.…”

E-Assessment Tools for Programming Languages: A Review

“…Four research questions were formulated to measure the degree to which e-assessment tools have helped to students and instructors [25]:…”

Juiz Online no ensino de Programação Introdutória - Uma Revisao Sistemática da Literatura

“…A -Existência de sistemas que sãoúteis para o ensino [Pettit et al 2015b], [Souza et al 2015] [Vihavainen et al 2013], [Petit et al 2012], [Llana et al 2012], [Sun et al 2014] 6 B -Redução da carga de trabalho do professor [Rajala et al 2016], [Wang et al 2011], [AlShamsi and Elnagar 2011] 3 C -Oferecer lista de exercícios extensa [Petit et al 2012] 1 D -Preocupações pedagógicas na criação dos problemas [Llana et al 2012], [Sun et al 2014], [Souza et al 2015] 3 E -Reduzir a ocorrência de plágio [Rajala et al 2016] 1 F -Aprendizagem no ritmo do aluno [Llana et al 2012] 1 G -Auto-aprendizagem [Wang et al 2011] 1 H -Motivação [Georgouli and Guerreiro 2010] 1 I -Melhorar a estratégia pedagógica [Rajala et al 2016], [Vihavainen et al 2013 A Tabela 5 relaciona os sistemas de Juiz Online mencionados nos artigos com os benefícios identificados na Tabela 4. A descrição *Experiência UL-FCIS foi incluída pelo fato do artigo não apresentar o nome do sistema, visto que o que aborda são as ideias de um protótipo produzido e os relatos da experiência.…”

“…O processo de submissão de programas ainda apresenta problemas, e o feedback nãoé suficiente para que alunos consigam corrigir muitos dos erros. F -Sistema contextualizado em competições [Georgouli and Guerreiro 2010], [Sun et al 2014] 2 G -Falta da informação sobre a contribuição dos sistemas para a aprendizagem [Pettit et al 2015b], [Georgouli and Guerreiro 2010], [Wang et al 2011] 3 H -Sistema aumentou ocorrência de plágio [Pozenel et al 2015] 1 I -Aluno não testar seus programas [Vihavainen et al 2013] 1…”