Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Abstract

Rater effects are commonly observed in rater-mediated assessments. By using item response theory (IRT) modeling, raters can be treated as independent factors that function as instruments for measuring ratees. Most rater effects are static and can be addressed appropriately within an IRT framework, and a few models have been developed for dynamic rater effects. Operational rating projects often require human raters to continuously and repeatedly score ratees over a certain period, imposing a burden on the cognitive processing abilities and attention spans of raters that stems from judgment fatigue and thus affects the rating quality observed during the rating period. As a result, ratees’ scores may be influenced by the order in which they are graded by raters in a rating sequence, and the rating order effect should be considered in new IRT models. In this study, two types of many-faceted (MF)-IRT models are developed to account for such dynamic rater effects, which assume that rater severity can drift systematically or stochastically. The results obtained from two simulation studies indicate that the parameters of the newly developed models can be estimated satisfactorily using Bayesian estimation and that disregarding the rating order effect produces biased model structure and ratee proficiency parameter estimations. A creativity assessment is outlined to demonstrate the application of the new models and to investigate the consequences of failing to detect the possible rating order effect in a real rater-mediated evaluation.

Rater Severity/Leniency and Bias in EFL Students' Composition Using Many-Facet Rasch Measurement (MFRM)

“…Before measuring rater severity, it is crucial to determine the item validity used as parameters for assessing students' writing. To ascertain this, the parameter utilized in the Many-Facet Rasch Measurement is the unidimensionality test (Sumintono & Widhiarso, 2013;Myford & Wolfe, 2004;Huang, 2023). The assumption test criteria used for testing the item's unidimensionality in the MFRM (Many-Facet Rasch Measurement) is the Raw Variance Explained by Measure, with a threshold value greater than 20% (≥ 20%); if it exceeds 40%, it indicates good quality, and if it surpasses 60%, it is considered exceptional (Sumintono & Widhiarso, 2013).…”

“…Thus, a comprehensive understanding of raters' tendencies towards being excessively severe or lenient is imperative to ensure fair and equitable evaluations of students' writing proficiency. Rater severity/leniency in students' composition refers to the degree to which raters consistently assign higher or lower scores when evaluating the written work of students (Myford & Wolfe, 2004;Huang, 2023). It is an important aspect of assessment in various educational contexts, including language education and writing courses.…”

“…Various factors can contribute to rater severity in students' composition. These factors include individual rater characteristics, such as personal biases, preferences, and experience levels (Fahim & Bijani, 2011;Huang, 2023;Noor, Beram, Huat, Gengatharan, & Mohamad Rasidi, 2023). Raters with strict personal standards may exhibit higher severity, while more lenient raters may display lower severity (Tanaka, 2023).…”