A Hierarchical Rater Model for Longitudinal Data

Abstract

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.

Investigation of Rater Tendencies and Reliability in Different Assessment Methods with Many Facet Rasch Model

“…In the literature, there are many studies conducted using the Many Facet Rasch Model Atılgan, 2005;Baştürk, 2010;Engelhard & Myford, 2003;Iramaneerart, Myford, Yudkowsky, & Lowenstein, 2009;Linacre et al, 1990;Nakamura, 2000;Nakamura, 2002) and directly examining the Many Facet Rasch Model (Casabianca & Junker, 2013, 2014DeCarlo 2010;DeCarlo et al, 2011;Iramaneerat, Yudkowsky, Myford, & Downing, 2008;Kim 2009;Lynch & McNamara, 1998;Mariano, 2002;Patz, Junker, & Johnson, 2000;Patz, Junker, Johnson, & Mariano, 2002;Verhelst & Verstralen, 2001;Wilson & Hoskens 2001). These studies are aimed at revealing the effect of the scoring category and the number of raters on the reliability of the measurements.…”

Accounting for Rater Effects With the Hierarchical Rater Model Framework When Scoring Simple Structured Constructed Response Tests

“…The RMSEs for rater bias ranged from roughly .001 to .34; for rater SD , RMSEs ranged from .001 to .27, with mostly an increase in RMSEs as the true value increased. While there appears no strict pattern, prior research has shown that the recovery of the HRM rater parameters are interdependent (relatively poor estimates of ϕ are yielded when ψ is small) and that for rater bias, recovery is typically best around discrete score levels (see Patz et al., and Casabianca, Junker, Nieto, & Bond, ). We see the latter phenomenon here at the middle of the scale (where bias ∼ 0).…”

“…We performed posterior predictive model checks (PPMC; Levy, Mislevy, & Sinharay, ; Sinharay, Johnson, & Stern, ) based on the M‐HRM to determine absolute fit of the model. Note that PPMC for the HRM has not been theoretically verified or tested with simulations, but has been applied to the longitudinal version of the model (Casabianca et al., ). For this example, we plotted the observed total score distributions for Speaking and Writing separately, with the total scores computed as the sum across item scores (see the thin solid lines in the top row of plots in Figure ).…”

A new item response theory model for rater centrality using a hierarchical rater model approach

“…First, when raters' evaluations extend over a long time, their errors may change across time (rater drift, Leckie & Baird, 2011). Moreover, latent traits might change or grow (Casabianca et al, 2017). Future studies can model this by extending HRM-SIC to multilevel models with time points nested within raters and students.…”