Automatic recognition of students' sorting algorithm implementations in a data structures and algorithms course

Abstract

Computing educators often rely on black-box analysis to assess students' work automatically and give feedback. This approach does not allow analyzing the quality of programs and checking if they implement the required algorithm. We introduce an instrument for recognizing and classifying algorithms (Aari ) in terms of white-box testing to identify authentic students' sorting algorithm implementations in a data structures and algorithms course. Aari uses machine learning techniques to classify new instances. The students were asked to submit a program to sort an array of integers in two rounds: at the beginning of the course before sorting algorithms were introduced, and after taking a lecture on sorting algorithms. We evaluated the performance of Aari with the implementations of each round separately. The results show that the sorting algorithms, which Aari has been trained to recognize, are recognized with an average accuracy of about 90%. When considering all the submitted sorting algorithm implementations (including the variations of the standard algorithms), Aari achieved an overall accuracy of 71% and 81% for the first and second round, respectively.In addition, we analyzed the students' implementations manually to gain a better understanding of the reasons of failure in the recognition process. This analysis revealed that students have many misconceptions related to sorting algorithms, which results in problematic implementations that are more inefficient compared with those of standard algorithms. We discuss these variations along with the application of the tool in an educational context, its limitations and some directions for future work.

E-Assessment Tools for Programming Languages: A Review

“…Also in 2012, Taherkhani [38] revealed that AARI (e-assessment tool) was successful at recognizing the algorithms used by the students to perform sorting on integers for about 75% of the submissions.Further in 2014, Gaudencio [39] stated that instructors who manually graded the assignments also tend to agree more with the feedback provided by the tool in comparison to assessment provided by other instructors.…”

Feature engineering for clustering student solutions

“…Huang et al [1] use unit test results and AST edit-distance algorithms to identify clusters of submissions that could potentially receive the same custom feedback message. Taherkhani et al [4] identify which sorting algorithm a student implemented using supervised machine learning methods. Each solution is represented by statistics about language constructs, measures of complexity, and detected roles of variables.…”

Toward facilitating assistance to students attempting engineering design problems

“…Tackling problems with multiple solutions directly, Taherkhani et al [6] demonstrated the practicality of differentiating between multiple solutions, i.e., different sorting algorithms, in students' solutions to a particular engineering design problem using a supervised machine learning method.…”