Limited‐information goodness‐of‐fit testing of hierarchical item factor models

Abstract

In applications of item response theory, assessment of model fit is a critical issue. Recently, limited-information goodness-of-fit testing has received increased attention in the psychometrics literature. In contrast to full-information test statistics such as Pearson’s X2 or the likelihood ratio G2, these limited-information tests utilise lower order marginal tables rather than the full contingency table. A notable example is Maydeu-Olivares and colleagues’ M2 family of statistics based on univariate and bivariate margins. When the contingency table is sparse, tests based on M2 retain better Type I error rate control than the full-information tests and can be more powerful. While in principle the M2 statistic can be extended to test hierarchical multidimensional item factor models (e.g., bifactor and testlet models), the computation is non-trivial. To obtain M2, a researcher often has to obtain (many thousands of) marginal probabilities, derivatives, and weights. Each of these must be approximated with high-dimensional numerical integration. We propose a dimension reduction method that can take advantage of the hierarchical factor structure so that the integrals can be approximated far more efficiently. We also propose a new test statistic that can be substantially better calibrated and more powerful than the original M2 statistic when the test is long and the items are polytomous. We use simulations to demonstrate the performance of our new methods and illustrate their effectiveness with applications to real data.

Assessing Approximate Fit in Categorical Data Analysis

“…Now, because M ord is a quadratic form of statistics that are a further reduction of the data than the statistics used in M 2 , from theory in Joe and Maydeu-Olivares (2010), (a) the empirical sample distribution of M ord is likely to be better approximated in small samples than the distribution of M 2 ; and (b) if (D ord /D 2 ) > 0.9, that is, if the ratio of noncentrality parameters for M ord and M 2 is sufficiently large, M ord will be more powerful than M 2 over a variety of alternative directions because there are fewer degrees of freedom associated with M ord than with M 2 . Cai and Hansen (2013) investigated the small sample distribution of M ord and M 2 in bifactor logistic models for polytomous data and reported that the sampling distribution of M ord is better approximated than that of M 2 when there are small expected counts in the bivariate tables. They also reported that M ord has higher power than M 2 to detect misspecified bifactor models.…”

“…We provide in the Appendix details on ord and ord for the graded IRT response model. Also, Cai and Hansen (2013) provided details 4 on how to compute these for bifactor IRT graded response models.…”

“…Interestingly, assessing the overall exact model fit in categorical data analysis had proved so difficult, except for very small models that are usually not of interest in applications, that the issue of goodness-of-fit has been outside the IRT research agenda for many years. Recently, there has been a growing interest in goodness-of-fit assessment in IRT (Bartholomew & Leung, 2001;Bartholomew & Tzamourani, 1999;Cai & Hansen, 2013;Cai, Maydeu-Olivares, Coffman, & Thissen, 2006;Glas, 1988;Glas & Verhelst, 1989, 1995, 2010Langeheine, Pannekoek, & van de Pol, 1996;Maydeu-Olivares, 2006;Maydeu-Olivares & Joe, 2005, 2006Reiser, 1996Reiser, , 2008Reiser & VandenBerg, 1994;Tollenaar & Mooijaart, 2003;Von Davier, 1997) and recent reviews (Mavridis, Moustaki, & Knott, 2007;Maydeu-Olivares, 2013;MaydeuOlivares & Joe, 2008) suggest that the issue of assessing whether IRT models fit exactly is well under way to being solved. Downloaded by [University of Sussex Library] at 06:37 09 February 2015…”

Summed Score Likelihood–Based Indices for Testing Latent Variable Distribution Fit in Item Response Theory

“…More general versions of the reduction operator matrices for multiple categorical IRT models can be derived using similar logic (see e.g., Maydeu-Olivares & Joe, 2006;Cai & Hansen, 2013). Note that L has full row rank.…”

“…In the simulation study M 2 will be used as a benchmark due to its numerous desirable properties identified in the literature (see e.g., Cai & Hansen, 2013). Performance of the proposed latent variable distribution fit indices will be evaluated against M 2 .…”