Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use

Abstract

In this study, we describe a framework for monitoring rater performance over time. We present several statistical indices to identify raters whose standards drift and explain how to use those indices operationally. To illustrate the use of the framework, we analyzed rating data from the 2002 Advanced Placement English Literature and Composition examination, employing a multifaceted Rasch approach to determine whether raters exhibited evidence of two types of differential rater functioning over time (i.e., changes in levels of accuracy or scale category use). Some raters showed statistically significant changes in their levels of accuracy as the scoring progressed, while other raters displayed evidence of differential scale category use over time.

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

“…First, only moderate stability of rater effects (r=0.6) was found across the two monitoring systems, somewhat worryingly suggesting that different impressions of rater performance could be given by the adoption of a particular system. Other studies too have shown instability in rater effects (Baird et al, 2013;Congdon & McQueen, 2000;Hoskens & Wilson, 2001;Harik et al, 2009;Lamprianou, 2006;Myford & Wolfe, 2009), which might be explained by small sample sizes in the monitoring checks.…”

“…One of our research questions was whether the different quality assurance systems would produce the same rank-ordering of raters' accuracy, as researchers have recently been interested in the stability of measures of rater effects over time or over subjects (e.g., Congdon & McQueen, 2000;Hoskens & Wilson, 2001;Harik et al, 2009;Lamprianou, 2006;Myford & Wolfe, 2009). The data included 22 examinations, with different styles of scoring rubric.…”

Risikotrekk og skjulte kvalitetar i elevtekstar

“…Eckes, 2012Eckes, , 2015Engelhard, 1994;Lumley, 2002;Myford & Wolfe, 2009;Weigle, 1998) eller vurdering relatert til tekstlingvistiske trekk (Freedman, 1979;Fritz & Ruegg, 2013;Wolfe, Song, & Jiao, 2016;Xie, 2015;Ö stlund-Stjärnegårdh, 2002). Vår studie har eit anna utgangspunkt: når oppgåver er sjangerlause, vil ein føresetnad for at vurderingane ikkje skal sprike, vere at ein møter tekstar med same lesemåte.…”

Application of Many-facet Rasch Model in In-basket Tests in China

“…Previous studies indicate that raters' severity or leniency can influence the rating process but their findings vary [2] [3] . With the advancement in personnel measurement, researchers turn to modern psychometrics to identify the root cause of errors and make adjustments to accommodate different types of errors.…”