Validity of peer grading using Calibrated Peer Review in a guided-inquiry, conceptual physics course

Abstract

Constructing and evaluating explanations are important science practices, but in large classes it can be difficult to effectively engage students in these practices and provide feedback. Peer review and grading are scalable instructional approaches that address these concerns, but which raise questions about the validity of the peer grading process. Calibrated Peer Review (CPR) is a web-based system that scaffolds peer evaluation through a "calibration" process where students evaluate sample responses and receive feedback on their evaluations before evaluating their peers. Guided by an activity theory framework, we developed, implemented, and evaluated CPR-based tasks in guided-inquiry, conceptual physics courses for future teachers and general education students. The tasks were developed through iterative testing and revision. Effective tasks had specific and directed prompts and evaluation instructions. Using these tasks, over 350 students at three universities constructed explanations or analyzed physical phenomena, and evaluated their peers' work. By independently assessing students' responses, we evaluated the CPR calibration process and compared students' peer reviews with expert evaluations. On the tasks analyzed, peer scores were equivalent to our independent evaluations. On a written explanation item included on the final exam, students in the courses using CPR outperformed students in similar courses using traditional writing assignments without a peer evaluation element. Our research demonstrates that CPR can be an effective way to explicitly include the science practices of constructing and evaluating explanations into large classes without placing a significant burden on the instructor.

Problematizing as a scientific endeavor

“…But we would like to incorporate more, and this is a place for pedagogical innovation. Perhaps calibrated peer review tools [67][68][69] can provide a practical way to implement such assignments in large classes.…”

Developing accomplished writers: Lessons from recent research

“…National assessments show that between two-thirds and three-fourths of students are themselves unskilled writers, and they may not be able to provide instructive feedback beyond simple grammar, spelling, and other surface-level issues. Peers are likely to provide useful support only when they are accomplished writers themselves, and when they have been taught some strategies for giving helpful criticism (Price et al, 2016).…”

Perspectives on peer-review: eight years of Aropä

“…Several articles demonstrate performance benefits through experience reports in different subjects: including, for example, nursing (Ramm et al, 2015), science writing (Walker & Sampson, 2013), biology, calculus and physiology (Reinholz, 2016), physics (Price et al, 2016) and instructional design (Brill, 2016).…”

“…His own critique of this document (or one written by a tutor) is made available to the students as soon as they have written their own review (using a 'locking' facility that allows students to lock-in their own review, removing the requirement to wait until the review deadline) so that they can compare it with their own review. A similar calibration process is described by Price et al (2016) whose students critique sample solutions and get feedback on their evaluations before evaluating their peers.…”