A multidimensional generalized many-facet Rasch model for rubric-based performance assessment

Abstract

Performance assessment, in which human raters assess examinee performance in a practical task, often involves the use of a scoring rubric consisting of multiple evaluation items to increase the objectivity of evaluation. However, even when using a rubric, assigned scores are known to depend on characteristics of the rubric’s evaluation items and the raters, thus decreasing ability measurement accuracy. To resolve this problem, item response theory (IRT) models that can estimate examinee ability while considering the effects of these characteristics have been proposed. These IRT models assume unidimensionality, meaning that a rubric measures one latent ability. In practice, however, this assumption might not be satisfied because a rubric’s evaluation items are often designed to measure multiple sub-abilities that constitute a targeted ability. To address this issue, this study proposes a multidimensional IRT model for rubric-based performance assessment. Specifically, the proposed model is formulated as a multidimensional extension of a generalized many-facet Rasch model. Moreover, a No-U-Turn variant of the Hamiltonian Markov chain Monte Carlo algorithm is adopted as a parameter estimation method for the proposed model. The proposed model is useful not only for improving the ability measurement accuracy, but also for detailed analysis of rubric quality and rubric construct validity. The study demonstrates the effectiveness of the proposed model through simulation experiments and application to real data.

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift

“…The MFRM is the most common type of model used for IRT with rater parameters (Linacre, 1989 ). Furthermore, there are various alternative models such as a two-parameter logistic model with rater severity parameters (Patz & Junker, 1999 ), generalized partial credit models incorporating various rater parameters (Uto, 2021b ; Uto & Ueno, 2020 ), hierarchical rater models (DeCarlo, Kim, & Johnson, 2011 ; Patz, Junker, Johnson, & Mariano, 2002 ; Qiu, Chiu, Wang, & Chen, 2022 ), extensions based on signal detection models (DeCarlo, 2005 ; Soo Park & Xing, 2019 ), rater bundle models (Wilson & Hoskens, 2001 ), and trifactor models (Shin et al, 2019 ). However, this study focuses on the MFRM because it is the most widely used and well-established of these models.…”

“…For complex models, however, it is not generally feasible to derive or calculate the marginal posterior distribution due to there being high-dimensional multiple integrals. MCMC, a random sampling-based estimation method, has been widely used in various fields to address this problem, including in IRT studies (Brooks, Gelman, Jones, & Meng, 2011 ; Fontanella et al, 2019 ; Fox, 2010 ; Uto, 2021b ; Uto & Ueno, 2020 ; van Lier et al, 2018 ; Zhang, Xie, You, & Huang, 2011 ).…”

“…It was recently developed along with a software package called “Stan” (Carpenter et al, 2017 ), which makes implementation of a NUT-based HMC easy. Thus, NUT has recently been widely used to perform parameter estimations for various statistical models, including IRT models (Jiang & Carter, 2019 ; Luo & Jiao, 2018 ; Uto, 2021b ; Uto & Ueno, 2020 ).…”

Learning Automated Essay Scoring Models Using Item-Response-Theory-Based Scores to Decrease Effects of Rater Biases

“…The unbiased essay scores θ j in the model can be estimated from observed essay rating data U while considering rater bias effects in a manner similar to that of the traditional GPCM, which can estimate examinee abilities while considering the effects of item characteristics. IRT models with rater parameters, including GMFRM, have been widely used for various performance tests, including essay writing tests and speaking tests, not only to realize an accurate ability or score estimation but also to analyze effects of various bias factors such as rater bias (e.g., [8]- [13], [35], [41]- [43], [58]).…”

Assessing proxies of knowledge and difficulty with rubric‐based instruments

“…In short, one cannot simply average the scores and call the output knowledge or learning. While some attempts have been made in the psychometric literature to measure knowledge with rubrics (e.g., Uto, 2021), to our knowledge, none have explicitly modeled the censoring problem. 1 In this paper, we show how to model and estimate this problem in a robust, understandable, way.…”