Automatic Grading of 3D Computer Animation Laboratory Assignments

Abstract

Assessment is a delicate task in the overall teaching process because it may require significant time and may be prone to subjectivity. Subjectivity is especially true for disciplines in which perceptual factors play a key role in the evaluation. In previous decades, computer-based assessment techniques were developed to address the above-mentioned challenges and to automatically grade students' work in a variety of educational fields. In this paper, the application of automatic assessment strategies in the unexplored domain of computer graphics is discussed. In particular, a tool that is designed to evaluate student assignments for a 3D computer animation course taught at Politecnico di Torino University is presented. During laboratory examination sessions, students are requested to individually operate on the open-source Blender suite and to recreate a 3D animation similar to a reference one. Student's output is assessed against a set of similarity indicators, which are specifically designed to capture the technical and perceptual factors that would be blended in a traditional teacher's evaluation. A comparison of the results that are achieved by the computer-based tool with grades assigned by visual inspection confirms the effectiveness of the designed approach and displays a high-quality concurrence between automatic and manual evaluations.

Auto-Grading for 3D Modeling Assignments in MOOCs

“…Automatic grading of 3D Modeling Assignments has been the focus of much research which has brought about development of tools in the fields of computer science, The conventional problem(s) associated with attaching a measurable label to a 3D Modeling assignment has been associated with the arrival at a formal metric to assess aesthetic value. While there has been research in this area, and a formal weighted metric defined by some universities that offer graduate courses in this domain such as the First School of Architecture of the Politecnico di Torino, Italy [6,7], this rule-based metric is difficult to implement in a general context especially in cases such as ours where creativity and imagination form a crucial step within the learning path. In these papers the authors present a specific subset of parameters that have to be adhered to in order for a submission to be graded.…”

Point Cloud-Based Automatic Assessment of 3D Computer Animation Courseworks

“…Despite the proliferation, in the computer science field, of software tools able to automatically identify errors in students' assignments and providing suitable feedback, only few of them dealt with the area of computer graphics [28]. A common strategy pursued in this domain to perform automatic assessment consists in analyzing the differences between a student's solution with respect to a reference.…”

“…As said, the alternative approach consists in analyzing raw data, i.e., data that can be obtained directly from the animation software without requiring (and depending on the results of) actual rendering. For instance, in [28], a strategy relying on the computation of a set of similarity indicators based on the position and orientation of joints used to animate 3D objects via armature deformation was presented. Despite the interesting results obtained, the main drawback of such approach is that it can only be exploited to assess animations created using the selected technique.…”

“…Hence, automatic assessment should be capable to deal with the largest number of animation techniques possible, in order to avoid the need to use different ad-hoc modules depending on the specific technique that has been adopted and enable consistent evaluations. Based on a review of related literature, in the last years the domain of automatic assessment in the area of computer graphics has been addressed through a limited, though significant, set of works, which considered the evaluation of hand sketches and other multimedia contents, as well as of the products of 2D and 3D modeling and animation tasks [23]- [28]. Unfortunately, methods developed so far to deal with 3D animated contents are not general-purpose, and have been tightly tailored to the specific software used (e.g., the free and open source Blender suite, like in [28]).…”

“…This choice allows the proposed method to be easily integrated in other assessment tools where global coordinates are considered. Similarly, it would be easy to generalize the proposed methodology to moving objects by simply taking into account also the metrics proposed, e.g., in [28].…”