Performance of Polytomous IRT Models With Rating Scale Data: An Investigation Over Sample Size, Instrument Length, and Missing Data

Abstract

The implementation of polytomous item response theory (IRT) models such as the graded response model (GRM) and the generalized partial credit model (GPCM) to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used. The performance of such models has not been fully investigated and compared across conditions with common survey-specific characteristics such as short test length, small sample size, and data missingness. The purpose of the current simulation study is to inform the literature and guide the implementation of GRM and GPCM under these conditions. For item parameter estimations, results suggest a sample size of at least 300 and/or an instrument length of at least five items for both models. The performance of GPCM is stable across instrument lengths while that of GRM improves notably as the instrument length increases. For person parameters, GRM reveals more accurate estimates when the proportion of missing data is small, whereas GPCM is favored in the presence of a large amount of missingness. Further, it is not recommended to compare GRM and GPCM based on test information. Relative model fit indices (AIC, BIC, LL) might not be powerful when the sample size is less than 300 and the length is less than 5. Synthesis of the patterns of the results, as well as recommendations for the implementation of polytomous IRT models, are presented and discussed.

Item parameter estimations for multidimensional graded response model under complex structures

“…In practice, the chances of obtaining accurate item parameter estimation based on the assumption that items are tightly restricted to perfect simple structure rather than multiple latent traits are uncertain (Finch, 2011). According to Dai et al (2021), the choice of polytomous IRT models (e.g., GRM and generalized partial credit model [GPCM]) is beyond the model fit indices, especially when the sample size is less than 300 and the test length is less than 5. Similarly, fitting a complex structure of multidimensionality to a simple structure may be inappropriate in practice.…”

“…Where P * jk (θ) is the probability that observed scores for item j and examinee i given the ability or latent trait θ to obtain a score greater than or equal to category k, D = 1 or 1.7, a jm is the vector of item discrimination parameters for item j on each latent trait m, b jk is the vector of item difficulty parameters for each category k within item j, θ m is the vector of the latent traits on m th dimension. However, the number of latent traits and category responses influence the dynamical feature of MGRM to GRM, and other multidimensional IRT models (e.g., multidimensional two-parameter logistic model; De Ayala, 1994;Embretson and Reise, 2000;Penfield, 2014;Dai et al, 2021).…”

Financial anxiety, financial self-efficacy, and general social supports: Reliability of assessments

“…The multidimensional graded response model, speci cally, has been recommended for survey assessment especially because of its ability to be used with lower sample sizes. 21 While very large sample sizes are not required to ensure generalizability, a large enough sample size is required to ensure accurate model t. 22 It has been recommended that sample sizes of at least 300 and instrument length of at least 5 be required for both GRM and GPCM models. 22 To determine which model resulted in the best t for the data, the M2 and SRMSR statistics were compared.…”

“…21 While very large sample sizes are not required to ensure generalizability, a large enough sample size is required to ensure accurate model t. 22 It has been recommended that sample sizes of at least 300 and instrument length of at least 5 be required for both GRM and GPCM models. 22 To determine which model resulted in the best t for the data, the M2 and SRMSR statistics were compared. 21,23 We also evaluated the models using Akaike and Bayesian Information Criterion (AIC and BIC).…”

Introduction and an application of the method of successive intervals: Focusing on scaling blameworthiness of behaviors

“…IRT 모형은 추정되는 모수에 따라 1 모수 모형 (Rasch model, Rasch, 1960), 2모수 모 형 (Birnbaum, 1958a(Birnbaum, , 1958bLord, 1952) 다 (Embretson & Reise, 2000). 반면에 연속적 간 격법은 통계적 가정에 크게 의존하지 않는다 (Rozeboom & Jones, 1956) (Dai et al, 2021, Reise & Yu, 1990 1952). 초기 연구들 (Guilford, 1954;Hevner, 1930;Saffir, 1937) (Dane, 1985;Dhami, 2008;Han & Park, 2017).…”