Robustness of Projective IRT to Misspecification of the Underlying Multidimensional Model

Abstract

As a method to derive a “purified” measure along a dimension of interest from response data that are potentially multidimensional in nature, the projective item response theory (PIRT) approach requires first fitting a multidimensional item response theory (MIRT) model to the data before projecting onto a dimension of interest. This study aims to explore how accurate the PIRT results are when the estimated MIRT model is misspecified. Specifically, we focus on using a (potentially misspecified) two-dimensional (2D)-MIRT for projection because of its advantages, including interpretability, identifiability, and computational stability, over higher dimensional models. Two large simulation studies (I and II) were conducted. Both studies examined whether the fitting of a 2D-MIRT is sufficient to recover the PIRT parameters when multiple nuisance dimensions exist in the test items, which were generated, respectively, under compensatory MIRT and bifactor models. Various factors were manipulated, including sample size, test length, latent factor correlation, and number of nuisance dimensions. The results from simulation studies I and II showed that the PIRT was overall robust to a misspecified 2D-MIRT. Smaller third and fourth simulation studies were done to evaluate recovery of the PIRT model parameters when the correctly specified higher dimensional MIRT or bifactor model was fitted with the response data. In addition, a real data set was used to illustrate the robustness of PIRT.

Using a Projection IRT Method for Vertical Scaling When Construct Shift Is Present

“…Results from the simulation indicate that, on average, both a * and d * were recovered well, across groups, even when the fitted 2D-MIRT was misspecified. This is consistent with the findings from Strachan et al (2020). For the curvilinearity condition, the item parameters for Group 2, on average, had the best recovery.…”

“…This is consistent with the findings from Strachan et al. (2020). For the curvilinearity condition, the item parameters for Group 2, on average, had the best recovery.…”

“…This is consistent with the observation that if the set of items all measure a similar linear combination of the underlying dimensions, then the data is essentially unidimensional (Reckase, 2010; Carlson, 2017). For the second issue, the 2‐dimensional MIRT for PIRT has been shown to be robust when the underlying data is of even higher dimensional as long as a strong primary dimension is present (Strachan et al., 2020). For strong construct shifts that may involve more than two dimensions (e.g., from substantive analysis of content), a higher‐than‐two dimensional MIRT may be required.…”

“…The purpose of fitting a 2D‐MIRT is that interpretation, indeterminacy, and identifiability issues are simpler to resolve in the 2D‐MIRT compared to higher‐dimensional models. Further, it has been shown that the 2D‐MIRT is rather robust to model misspecification (Strachan et al., 2020). In the following analyses, all PIRT fitted a 2D‐MIRT for projection.…”

Several Variations of Simple‐Structure MIRT Equating

“…For instance, for data projected onto five dimensions, the use of 41 quadrature points for each dimension leads to 41 5 (=115,856,201) combinations of theta values to be evaluated. One possible option to ease this complexity is to specify the number of quadrature points fewer than 41, which has been commonly adopted in several studies (DeMars, 2013; Strachan et al., 2020; van Lier et al., 2018). However, using too few quadrature points may lower the accuracy of estimation due to the rough approximation to represent the ability distribution.…”