Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Park, Yoon Soo; Xing, Kuan

doi:10.1080/00273171.2018.1522496

Cited by 6 publications

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MFRM is the most common type of model used for IRT with rater parameters (Linacre, 1989 ). Furthermore, there are various alternative models such as a two-parameter logistic model with rater severity parameters (Patz & Junker, 1999 ), generalized partial credit models incorporating various rater parameters (Uto, 2021b ; Uto & Ueno, 2020 ), hierarchical rater models (DeCarlo, Kim, & Johnson, 2011 ; Patz, Junker, Johnson, & Mariano, 2002 ; Qiu, Chiu, Wang, & Chen, 2022 ), extensions based on signal detection models (DeCarlo, 2005 ; Soo Park & Xing, 2019 ), rater bundle models (Wilson & Hoskens, 2001 ), and trifactor models (Shin et al, 2019 ). However, this study focuses on the MFRM because it is the most widely used and well-established of these models.…”

Section: Many-facet Rasch Models For Rater Severity Driftmentioning

confidence: 99%

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

Uto

2022

Behav Res

View full text Add to dashboard Cite

Fair performance assessment requires consideration of the effects of rater severity on scoring. The many-facet Rasch model (MFRM), an item response theory model that incorporates rater severity parameters, has been widely used for this purpose. Although a typical MFRM assumes that rater severity does not change during the rating process, in actuality rater severity is known to change over time, a phenomenon called rater severity drift. To investigate this drift, several extensions of the MFRM have been proposed that incorporate time-specific rater severity parameters. However, these previous models estimate the severity parameters under the assumption of temporal independence. This introduces inefficiency into the parameter estimation because severities between adjacent time points tend to have temporal dependency in practice. To resolve this problem, we propose a Bayesian extension of the MFRM that incorporates time dependency for the rater severity parameters, based on a Markov modeling approach. The proposed model can improve the estimation accuracy of the time-specific rater severity parameters, resulting in improved estimation accuracy for the other rater parameters and for model fitting. We demonstrate the effectiveness of the proposed model through simulation experiments and application to actual data.

show abstract

Section: Many-facet Rasch Models For Rater Severity Driftmentioning

confidence: 99%

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

Uto

2022

Behav Res

View full text Add to dashboard Cite

show abstract

A Multidimensional Item Response Theory Model for Rubric-Based Writing Assessment

Uto

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A multidimensional generalized many-facet Rasch model for rubric-based performance assessment

Uto

2021

Behaviormetrika

View full text Add to dashboard Cite

Performance assessment, in which human raters assess examinee performance in a practical task, often involves the use of a scoring rubric consisting of multiple evaluation items to increase the objectivity of evaluation. However, even when using a rubric, assigned scores are known to depend on characteristics of the rubric’s evaluation items and the raters, thus decreasing ability measurement accuracy. To resolve this problem, item response theory (IRT) models that can estimate examinee ability while considering the effects of these characteristics have been proposed. These IRT models assume unidimensionality, meaning that a rubric measures one latent ability. In practice, however, this assumption might not be satisfied because a rubric’s evaluation items are often designed to measure multiple sub-abilities that constitute a targeted ability. To address this issue, this study proposes a multidimensional IRT model for rubric-based performance assessment. Specifically, the proposed model is formulated as a multidimensional extension of a generalized many-facet Rasch model. Moreover, a No-U-Turn variant of the Hamiltonian Markov chain Monte Carlo algorithm is adopted as a parameter estimation method for the proposed model. The proposed model is useful not only for improving the ability measurement accuracy, but also for detailed analysis of rubric quality and rubric construct validity. The study demonstrates the effectiveness of the proposed model through simulation experiments and application to real data.

show abstract

Rater Model Using Signal Detection Theory for Latent Differential Rater Functioning

Cited by 6 publications

References 19 publications

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

A Multidimensional Item Response Theory Model for Rubric-Based Writing Assessment

A multidimensional generalized many-facet Rasch model for rubric-based performance assessment

Contact Info

Product

Resources

About