Bias and Bias Correction Method for Nonproportional Abilities Requirement (NPAR) Tests

Applied Psychological Measurement

Fu³

et al. 2020

Self Cite

As a method to derive a “purified” measure along a dimension of interest from response data that are potentially multidimensional in nature, the projective item response theory (PIRT) approach requires first fitting a multidimensional item response theory (MIRT) model to the data before projecting onto a dimension of interest. This study aims to explore how accurate the PIRT results are when the estimated MIRT model is misspecified. Specifically, we focus on using a (potentially misspecified) two-dimensional (2D)-MIRT for projection because of its advantages, including interpretability, identifiability, and computational stability, over higher dimensional models. Two large simulation studies (I and II) were conducted. Both studies examined whether the fitting of a 2D-MIRT is sufficient to recover the PIRT parameters when multiple nuisance dimensions exist in the test items, which were generated, respectively, under compensatory MIRT and bifactor models. Various factors were manipulated, including sample size, test length, latent factor correlation, and number of nuisance dimensions. The results from simulation studies I and II showed that the PIRT was overall robust to a misspecified 2D-MIRT. Smaller third and fourth simulation studies were done to evaluate recovery of the PIRT model parameters when the correctly specified higher dimensional MIRT or bifactor model was fitted with the response data. In addition, a real data set was used to illustrate the robustness of PIRT.

mentioning

confidence: 99%

Robustness of Projective IRT to Misspecification of the Underlying Multidimensional Model

Applied Psychological Measurement

Fu³

et al. 2020

Self Cite

“…Specific patterns of item structures may lead to such nonlinearity. Ip et al (2019) provided an example of such condition, which was termed non-proportional ability requirement (NPAR). Using PISA data as an example, the authors described a situation under which disproportionate cognitive demand across the dimensions is required for items that reside at certain regions (e.g., high ends) of the latent abilities.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of the Linear Composite Conjecture for Unidimensional IRT Scale for Multidimensional Responses

Applied Psychological Measurement

Cho

Ackerman

et al. 2022

Self Cite

The linear composite direction represents, theoretically, where the unidimensional scale would lie within a multidimensional latent space. Using compensatory multidimensional IRT, the linear composite can be derived from the structure of the items and the latent distribution. The purpose of this study was to evaluate the validity of the linear composite conjecture and examine how well a fitted unidimensional IRT model approximates the linear composite direction in a multidimensional latent space. Simulation experiment results overall show that the fitted unidimensional IRT model sufficiently approximates linear composite direction when correlation between bivariate latent variables is positive. When the correlation between bivariate latent variables is negative, instability occurs when the fitted unidimensional IRT model is used to approximate linear composite direction. A real data experiment was also conducted using 20 items from a multiple-choice mathematics test from American College Testing.

“…Essentially, this means that the dimensional structure of the items changes among the different ability groups. For the current simulation study, we adopted a similar idea by confounding dimensionality and difficulty; that is, as the test items become more difficult in each successive grade, the dimensionality of the test items changed (Ip et al., 2019). We also included a condition where no curvilinearity is presented among the groups.…”

Section: Relationship Of Mirt and Locally Dependent Irtmentioning

confidence: 99%

Using a Projection IRT Method for Vertical Scaling When Construct Shift Is Present

J Educational Measurement

Cho

Kim

et al. 2020

Self Cite

In vertical scaling, results of tests from several different grade levels are placed on a common scale. Most vertical scaling methodologies rely heavily on the assumption that the construct being measured is unidimensional. In many testing situations, however, such an assumption could be problematic. For instance, the construct measured at one grade level may differ from that measured in another grade (e.g., construct shift). On the other hand, dimensions that involve low‐level skills are usually mastered by almost all students as they progress to higher grades. These types of changes in the multidimensional structure, within and across grades, create challenges for developing a vertical scale. In this article, we propose the use of projective IRT (PIRT) as a potential solution to the problem. Assuming that a test measures a primary dimension of substantive interest as well as some peripheral dimensions, the idea underlying PIRT is to integrate out the secondary dimensions such that the model provides both item parameters and ability estimates for the primary dimension. A simulation study was conducted to evaluate the effectiveness of the PIRT as a method for vertical scaling. An example using empirical data from a measure of foundational reading skills is also presented.