Examining The Rater Drift in The Assessment of Presentation Skills in Secondary School Context

Abstract

Among the sub-categories of assessment types, alternative assessment including peer-assessment helps students develop metacognition. The reliability and validity issues are the most significant problem in peer-assessment. Furthermore, the studies focusing on rater bias in peer review are limited. This study investigated the rater severity drift, which is one of the rater effects, in peer-assessment. The performance of 8 oral presentations based on group work in Science and Technology course was scored by 7th grade students (N=28) by using the rubric that researchers developed. The presentations lasted for four days with two presentations in each day. While examining the time-dependent drift in rater severity in peer-assessment, the many-Facet Rasch Measurement model was used. Two indexes (interaction term and standardized differences) were calculated with many-Facet Rasch Measurement with an aim to determine whether the raters made rater severity drift on a group or an individual basis. The analysis examined the variance of scores in following days in comparison with the scores of the 1st day. Accordingly, the two methods used to determine rater severity shift gave similar results, and some raters at the individual level tended to be more strict or lenient over time although there was no significant rater drift at the group level, showing that drifts had no specific models.

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

“…Valid measurements of a student's performance through scoring by different raters can only be achieved if the scoring is reliable (Hafner & Hafner, 2003;Farrokhi et al, 2011;Nalbantoglu Yilmaz, 2017). However, various factors related to raters can mix with measurement results when evaluating a student's performance (Erman-Aslanoglu & Sata, 2023). These factors related to raters that affect a student's performance are referred to as the rater effect (Farrokhi et al, 2011).…”

“…In the literature, there are various sources of error related to raters. The most commonly encountered ones in the literature are scorer severity and leniency, the halo effect, central tendency bias, and rater drift (Dogan & Uluman, 2017;Hoyt, 2000;Szőkol et al, 2022;Erman-Aslanoglu & Sata, 2023;Wolfe et al, 2007).…”

“…The Data analysis stage used the many-Facet Rasch Measurement (MFRM) approach. MFRM is recommended for determining the reliability and validity of self, peer, and teacher assessment ratings (Erman-Aslanoglu & Sata, 2023;Uto, 2022). It can also be said that MFRM proposes various extensions to investigate the phenomena of time-specific rater severity, also known as rater drift (Uto, 2022).…”