A factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments

Abstract

Evaluating open-response assignments in Massive Open Online Courses is a difficult task because of the huge number of students involved. Peer grading is an effective method to address this problem. There are two basic approaches in the literature: cardinal and ordinal. The first case uses grades assigned by student-graders to a set of assignments of other colleagues. In the ordinal approach, the raw materials used by grading systems are the relative orders that graders appreciate in the assignments that they evaluate. In this paper we present a factorization method that seeks a trade-off between cardinal and ordinal approaches. The algorithm learns from preference judgments to avoid the subjectivity of the numeric grades. But in addition to preferences expressed by student-graders, we include other preferences: those induced from assignments with significantly different average grades. The paper includes a report of the results obtained using this approach in a real world dataset collected in 3 Universities of Spain, A Coruña, Pablo de Olavide at Sevilla, and Oviedo at Gijón. Additionally, we studied the sensitivity of the method with respect to the number of assignments graded by each student. Our method achieves similar or better scores than staff instructors when we measure the discrepancies with other instructor's grades. * Corresponding author: Tel: +34 985 182 028

Including Content-Based Methods in Peer-Assessment of Open-Response Questions

“…We adopt the approach called peer-assessment [1], [2], [3], [4], [5], [6], [7], [8] as the basic strategy. The students are asked to evaluate a small set of anonymized assignments submitted by other students.…”

Content-based methods in peer assessment of open-response questions to grade students as authors and as graders

“…This research is an extension of research described in two conference papers [17,16], but now with a considerably broader experimental setting. Moreover, the performance of our method has been considerably enhanced over that described in previous research that did not use answer content [7].…”

“…We tackled the challenge of evaluating open-response questions, adopting, as our basic strategy, peer assessment [1,2,3,4,5,6,7,8], whereby students evaluate the anonymized answers of other students participating in the same course. Students, in their role as graders, receive a set of detailed rules (called a rubric) designed to ensure uniform assessment.…”

“…In the ordinal approach to peer assessment, graders rank answers in terms of their quality [5,8,7] -clearly an easier task for inexpert graders than cardinal grading, as evidenced by the considerably higher reliability reported for ordinal compared to cardinal assessment [9,10,11,1,12]. (See [13] for an interesting discussion of cardinal and ordinal peer grading from a psychological point of view).…”

“…We use a factorization method to train a utility function that estimates consensus in rankings of answers. This approach -inspired by a preference learning framework [15,9] -was used in previous research by us [6,7,16,17]. Answers can be represented by vectors of features, which have been acknowledged to be crucial for the success of peer assessment [18,19].…”