The Recent Development of Automated Programming Assessment

Abstract

Nowadays computer programs are objectively tested and marked by automated programming assessment systems in computer science education. Dynamic analysis and static analysis are two major approaches in the field of automated programming assessment. Methods adopted by dynamic and static analysis have been emphasized and detailed in this paper. Advantages and disadvantages of them implemented in instruction have been analyzed when taking instructional practice into consideration. Although many automated programming assessment systems have been proved to be of great help to both instructors and students in programming instruction, several problems remain unsolved, such as the security problem and algorithms for automatic generation of test data in dynamic analysis, low accuracy and precision of correctness and functionality assessment in static analysis, thus optimal approaches are still under research. The way of effective use of dynamic analysis and static analysis in instructional practice is also suggested in order that instructors can use the results of this study to choose the most appropriate approach in the context of a particular instructional goal. Finally, standardization of automated programming assessment systems, way to open source code and intelligent tutor system for automated programming assessment are expected to come into exist in the future.

Semi-Automatic Assessment of Unrestrained Java Code

“…In this area, almost all approaches are based on output comparison (see, e.g., [13,5], and the four surveys [2,1,10,7]). The general idea is to compile the students code, execute it with a predetermined set of inputs, and compare the outputs with the expected results.…”

The effect of automatic assessment on novice programming: Strengths and limitations of existing systems

“…The literature shows that little work has been done on the survey of AA systems for programming codes [2,20,28,48], although a tremendous number of tools and systems have been developed. Most of the surveyed papers have characterized the AA systems for programming codes into two major approaches: dynamic and static analysis [2,28]. In the dynamic analysis approach, the assessment depends on program execution, while it is not in the static analysis approach [2].…”

“…[2]; others surveyed the AA systems that are using only the static analysis approaches [48] and more as in Ref. [28]. Reference [10] covers the systems developed in a certain period from 1960 to 2005 and Ref.…”

“…This paper will use the same categorization as used by Ala-Mutka [2] and Liang et al [28], in addition to a hybrid approach, to demonstrate the existing systems from the literature. Figure 1 shows the AA systems taxonomy based on analysis approaches, where the hybrid approach combines both approaches: static and dynamic analysis.…”

“…Static analysis is intended to assess the program code without executing it [2,26], which means it is achievable even if the code's compilation fails [28]. Static analysis is mainly based on similarity of the code with the model code [25], in addition to some measurable criteria such as the total number of variables, statements, expressions, etc., and symbolic execution [48,51], which is used consequently to assess the correctness, efficiency [28], plagiarism [37,51], keywords matching, diagram analysis [26], and quality of the code [28]. Correctness means that the code is syntactically correct, which can be easily checked through a compiler or interpreter [2].…”