Assessing Approximate Fit in Categorical Data Analysis

Abstract

A family of Root Mean Square Error of Approximation (RMSEA) statistics is proposed for assessing the goodness of approximation in discrete multivariate analysis with applications to item response theory (IRT) models. The family includes RMSEAs to assess the approximation up to any level of association of the discrete variables. Two members of this family are RMSEA2, which uses up to bivariate moments, and the full information RMSEAn. The RMSEA2 is estimated using the M2 statistic of Maydeu-Olivares and Joe (2005, 2006), whereas for maximum likelihood estimation, RMSEAn is estimated using Pearson's X(2) statistic. Using IRT models, we provide cutoff criteria of adequate, good, and excellent fit using the RMSEA2. When the data are ordinal, we find a strong linear relationship between the RMSEA2 and the Standardized Root Mean Squared Residual goodness-of-fit index. We are unable to offer cutoff criteria for the RMSEAn as its population values decrease as the number of variables and categories increase.

Identifying the Source of Misfit in Item Response Theory Models

“…A 90% confidence interval for the RMSEA 2 yields [0.03, 0.04]. MaydeuOlivares and Joe (2014) suggest that IRT models with an RMSEA 2 less than or equal to 0.05 provide a close approximation to the data-generating model and that those with an RMSEA 2 less than or equal to 0.05 / (K -1) provide an excellent approximation. Because K = 5, their criterion for an excellent approximation is RMSEA 2 ≤ 0.0125.…”

“…Our advice in this application is to attempt to find a better fitting model. Failing to do so, the fitted model may be used as it provides a close fit to the data using the criteria of Maydeu-Olivares and Joe (2014). Here is a word of caution: a piecewise assessment shall be performed regardless of the value of the RMSEA 2 (or similar overall measure of fit).…”

Evaluating Structural Equation Models for Categorical Outcomes: A New Test Statistic and a Practical Challenge of Interpretation

“…For example, Maydeu-Olivares (2013) developed a rationale for constructing an M 2 -based RMSEA. More recently, Maydeu-Olivares & Joe (2014) expanded on this line of research and proposed some cutoff criteria for approximate fit. Another example is provided by Lee and Cai (2012), which proposed an M 2 -based Tucker-Lewis Index (Tucker & Lewis, 1973).…”

Can We Learn From Student Mistakes in a Formative, Reading Comprehension Assessment?

“…Given the conceptual partial ordering and the test option response functions shown in Figure , we decided to fit both ordered and nominal models for polytomous data. The following decision rule was adopted regarding fit: on the basis of the AIC and BIC, select the model that performs best across all forms and grades from among those with acceptable RMSEAs (RMSEA ≤ 0.09; Browne & Cudeck, ; Hu & Bentler, ; Maydeu‐Olivares & Joe, ). Also, we preferred to use the same model for all forms.…”