Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects

Abstract

Peer grading offers a scalable and sustainable way of providing assessment and feedback to a massive student population. However, currently there is little empirical evidence to support the credentials of peer grading as a learning assessment method in the MOOC context. To address this research need, this study examined 1,825 peer grading assignments collected from a Coursera MOOC with the purpose of investigating the reliability and validity of peer grading, as well as its perceived effects on students' MOOC learning experience. The empirical findings provide evidence that the aggregate of student graders can provide peer grading scores fairly consistent and highly similar to instructor grading scores. Student survey responses also indicate peer grading activities to be well received by a majority of MOOC students, who believe it was fair, useful, beneficial, and would recommend it to be included in future MOOC offerings. Based on the empirical results, this study concludes with a set of principles for designing and implementing peer grading activities in the MOOC context.

On the validity of peer grading and a cloud teaching assistant system

“…The measure of reliability most commonly studied is inter-rater reliability, which refers to the consistency of grades that different peers would assign to the same assessment. Validity concepts assume that the instructor provides a trustworthy grade, and hence validity of peer grading is calculated as a correlation coefficient between peer grades and instructor grades [5,15,3]. Examples of reported reasons to doubt both the reliability and validity of peer grading include lack of experience with the required content and methods, as well as bias due to uniformity, race, friendship [6] or even manipulation [16].…”

The educational problem that MOOCs could solve: professional development for teachers of disadvantaged students

“…Studies of peer assessment methods used in MOOCs have emphasised the importance of using qualitative assessment which recipients can use as guidance for improving their work (Carless et al 2010), and of guiding their assessment by defining criteria and scoring rubrics to improve the validity (Luo, Robinson, and Park 2014). For our teacher participants, the assignments asked them to develop documents, plans or resources for their future work beyond the course.…”

Paragogy and flipped assessment: experience of designing and running a MOOC on research methods

“…The discussion surrounding it, however, has been largely fixated on the extent to which peer grading results could match up with grades by experts and how the former could be brought closer to the latter (e.g. Luo, Robinson, & park, 2014;piech, Huang, Chen, ng, & Koller, 2013). In the realm of research, this is problematic.…”