Investigating the Impact of Item Parameter Drift for Item Response Theory Models with Mixture Distributions

Abstract

This study investigates the impact of item parameter drift (IPD) on parameter and ability estimation when the underlying measurement model fits a mixture distribution, thereby violating the item invariance property of unidimensional item response theory (IRT) models. An empirical study was conducted to demonstrate the occurrence of both IPD and an underlying mixture distribution using real-world data. Twenty-one trended anchor items from the 1999, 2003, and 2007 administrations of Trends in International Mathematics and Science Study (TIMSS) were analyzed using unidimensional and mixture IRT models. TIMSS treats trended anchor items as invariant over testing administrations and uses pre-calibrated item parameters based on unidimensional IRT. However, empirical results showed evidence of two latent subgroups with IPD. Results also showed changes in the distribution of examinee ability between latent classes over the three administrations. A simulation study was conducted to examine the impact of IPD on the estimation of ability and item parameters, when data have underlying mixture distributions. Simulations used data generated from a mixture IRT model and estimated using unidimensional IRT. Results showed that data reflecting IPD using mixture IRT model led to IPD in the unidimensional IRT model. Changes in the distribution of examinee ability also affected item parameters. Moreover, drift with respect to item discrimination and distribution of examinee ability affected estimates of examinee ability. These findings demonstrate the need to caution and evaluate IPD using a mixture IRT framework to understand its effects on item parameters and examinee ability.

Practical Consequences of Item Response Theory Model Misfit in the Context of Test Equating with Mixed-Format Test Data

“…Furthermore, once a certain IRT model is determined in the beginning of the administration of an assessment program, such a model typically has to be sustained for item calibration, test equating and scoring in the subsequent administrations from year to year. The degree of misfit could be varied across administrations due to reasons like changes of latent trait distributions and item parameter drift (Park et al, 2016). Depending on the intended model uses, the practical consequences of misfit might or might not be tolerable.…”

A Study on the Identification of Latent Classes Using Mixture Item Response Theory Models: TIMSS 2015 Case

“…Standard IRT models are used for calibration of item parameters and scaling of individual performances in international large-scale assessments such as TIMSS and PISA (Martin et al, 2016). Literature review revealed that latent classes are ignored at the end of the analyses conducted with IRT models in some studies that employed international large-scale test data (Kreiner & Christensen, 2014;Oliveri & von Davier, 2011;Oliveri & von Davier, 2014;Park et al, 2016). In the presence of latent classes, the parameter invariance assumption of standard IRT models is violated and biased results can be obtained in item parameter calibrations (DeMars & Lau, 2011).…”

“…These results indicate the presence of latent classes in the data of countries with high, medium and low performance regardless of country's performance ranking. The parameter invariance assumption, which is one of the assumptions of standard IRT models, is violated in the presence of latent classes (Park et al,2016). As the parameter invariance assumption could not be met, it was concluded that the data obtained from the 8th grade science subtest of TIMSS 2015 fit Mixture IRT models.…”

“…Analysis of the relevant literature reveals that there are many studies examining the existence of latent classes in international large-scale test data (Choi, Alexeev, &Cohen, 2015;Liu, Liu, & Li, 2018;Oliveri, Ercikan, Zumbo, & Lawless, 2014;Oliveri & von Davier, 2011;Park, Lee, & Xing, 2016;Sen, Cohen & Kim, 2016;Toker, 2016;Zhang, Orrill, & Campbell, 2015). Three latent classes were identified in a study examining the heterogeneity in response patterns of fourth-grade students from Taiwan, Hong Kong, Qatar and Kuwait who participated in PIRLS 2006 .…”

“…In this situation biased results can be obtained in item parameter calibrations (DeMars & Lau, 2011). Furthermore, although it is emphasized that the parameter invariance assumption is necessary for cross-cultural comparisons (Hambleton & Rogers, 1989;Meredith, 1993;Millsap, 2011), testing of assumptions for data obtained from international assessments can be neglected (Park et al, 2016).…”