Identifying the Source of Misfit in Item Response Theory Models

Liu, Yang; Maydeu-Olivares, Alberto

doi:10.1080/00273171.2014.910744

Cited by 18 publications

(15 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Drasgow et al (1995) suggested adjusting the test statistic and also proposed a critical value for it, which they determined with Monte Carlo methods. Reiser (1996), , and Liu and Maydeu-Olivares (2014) finally derived the large sample distribution of the difference between the observed and expected frequencies in single cross tabulations. Liu and Maydeu-Olivares (2012) generalized the S − X 2 -test of Orlando and Thissen (2003) to item pairs.…”

Section: Tests Of Model Fitmentioning

confidence: 99%

Analyzing the Fit of IRT Models With the Hausman Test

Ranger

Much

2020

Front. Psychol.

View full text Add to dashboard Cite

In this manuscript, the applicability of the Hausman test to the evaluation of item response models is investigated. The Hausman test is a general test of model fit. The test assesses whether for a model in question the parameter estimates of two different estimators coincide. The test can be implemented for item response models by comparing the parameter estimates of the marginal maximum likelihood estimator with the corresponding parameter estimates of a limited information estimator. For a correctly specified item response model, the difference of the two estimates is normally distributed around zero. The Hausman test can be used for the evaluation of item fit and global model fit. The performance of the test is evaluated in a simulation study. The simulation study suggests that the implemented versions of the test adhere to the nominal Type-I error rate well in samples of 1000 test takers and more. The test is also capable to detect misspecified item characteristic functions, but lacks power to detect violations of the conditional independence assumption. Keywords: item response theory, 2-PL model, model fit, item fit, Hausman test ANALYZING THE FIT OF IRT MODELS WITH THE HAUSMAN TESTItem response models are measurement models that allow inferring individual traits from responses given to the items of a standardized test. Core of item response models are precise assumptions about the relation between the traits and the response in a single item and the interrelation of the responses from different items. These assumptions then serve as a mathematical basis for deducing statements about a test taker's traits from his/her responses. The correctness of such inferential statements depends crucially on the validity of the item response model. As the results of psychological assessment often have important consequences, one has to guarantee that the conclusions drawn about the test takers have a sound basis. Therefore, it is indispensable to check the adequacy of the chosen item response model and its assumptions carefully. Such a check requires a powerful test of model fit.Several tests of model fit have been proposed in the past. A short overview over of the different tests is given in the following section. In doing so, the focus is mainly on the two-parameter logistic model. Nothing will be said about tests that were proposed exclusively for the Rasch model and cannot be used in general; for such tests see Glas and Verhelst (1995), Suaréz Falcón and Glas (2003), and Maydeu-Olivares and Montaño (2013. The review does also not cover the general approaches used in non-parametric item response theory (Sijtsma, 1998) or tests within the Bayesian framework (Sinharay, 2016). Tests of differential item functioning will also not be addressed (Magis et al., 2010). Having given this overview, an alternative test of model fit is Frontiers in Psychology | www.frontiersin.org 1 February 2020 | Volume 11 | Article 149 Ranger and Much Hausman Test for IRT Models Frontiers in Psychology | www.frontiersin.org

show abstract

Section: Tests Of Model Fitmentioning

confidence: 99%

Analyzing the Fit of IRT Models With the Hausman Test

Ranger

Much

2020

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…In future studies, this issue can be investigated. We also believe that these new information estimation methods will play important roles in other applications of CDMs, for example to identify locally dependent item pairs in IRT modelling (Liu & Maydeu‐Olivares, , ; Liu & Thissen, ).…”

Section: Discussionmentioning

confidence: 99%

Information matrix estimation procedures for cognitive diagnostic models

Liu

Xin

Andersson

et al. 2018

Brit J Math & Statis

View full text Add to dashboard Cite

Two new methods to estimate the asymptotic covariance matrix for marginal maximum likelihood estimation of cognitive diagnosis models (CDMs), the inverse of the observed information matrix and the sandwich-type estimator, are introduced. Unlike several previous covariance matrix estimators, the new methods take into account both the item and structural parameters. The relationships between the observed information matrix, the empirical cross-product information matrix, the sandwich-type covariance matrix and the two approaches proposed by de la Torre (2009, J. Educ. Behav. Stat., 34, 115) are discussed. Simulation results show that, for a correctly specified CDM and Q-matrix or with a slightly misspecified probability model, the observed information matrix and the sandwich-type covariance matrix exhibit good performance with respect to providing consistent standard errors of item parameter estimates. However, with substantial model misspecification only the sandwich-type covariance matrix exhibits robust performance.

show abstract

“…Actually, the model-data fit at the global model level must be investigated firstly using model fit indices, when a model does not fit well, alternative models might be fitted. However, more often than not, no such model provides a good fit (Liu & Maydeu-Olivares, 2014). Facing this situation, researchers have to differentiate well-fitting items from poorly fitting ones; then they may decide to retain only the well-fitting set or to apply an alternative IRT model to the poorly fitting set on the basis of item fit analysis.…”

Section: Discussionmentioning

confidence: 99%

“…On the contrary, the limited information fit statistics (Bartholomew & Leung, 2002; Cai, Maydeu-Olivares, Coffman, & Thissen, 2006; Maydeu-Olivares & Joe, 2005; Reiser, 2008) used the marginal tables (i.e., the cross tabulations of item pairs or item triplets), rather than frequencies of single response patterns in chi-square-based item fit statistics to identify misfit. These sets of indices were often used to assess item fit in sparse contingency tables (Cai et al, 2006; Maydeu-Olivares & Joe, 2005), detect local independence (Liu & Maydeu-Olivares, 2013), and identify the source of misfit (Liu & Maydeu-Olivares, 2014).…”

Section: Introductionmentioning

confidence: 99%

Assessing Item-Level Fit for Higher Order Item Response Theory Models

Zhang

Wang

Tao

2018

Applied Psychological Measurement

View full text Add to dashboard Cite

Testing item-level fit is important in scale development to guide item revision/deletion. Many item-level fit indices have been proposed in literature, yet none of them were directly applicable to an important family of models, namely, the higher order item response theory (HO-IRT) models. In this study, chi-square-based fit indices (i.e., Yen’s Q1, McKinley and Mill’s G2, Orlando and Thissen’s S-X2, and S-G2) were extended to HO-IRT models. Their performances are evaluated via simulation studies in terms of false positive rates and correct detection rates. The manipulated factors include test structure (i.e., test length and number of dimensions), sample size, level of correlations among dimensions, and the proportion of misfitting items. For misfitting items, the sources of misfit, including the misfitting item response functions, and misspecifying factor structures were also manipulated. The results from simulation studies demonstrate that the S-G2 is promising for higher order items.

show abstract

Identifying the Source of Misfit in Item Response Theory Models

Cited by 18 publications

References 46 publications

Analyzing the Fit of IRT Models With the Hausman Test

Analyzing the Fit of IRT Models With the Hausman Test

Information matrix estimation procedures for cognitive diagnostic models

Assessing Item-Level Fit for Higher Order Item Response Theory Models

Contact Info

Product

Resources

About