On the Choice of the Item Response Model for Scaling PISA Data: Model Selection Based on Information Criteria and Quantifying Model Uncertainty

Abstract

In educational large-scale assessment studies such as PISA, item response theory (IRT) models are used to summarize students’ performance on cognitive test items across countries. In this article, the impact of the choice of the IRT model on the distribution parameters of countries (i.e., mean, standard deviation, percentiles) is investigated. Eleven different IRT models are compared using information criteria. Moreover, model uncertainty is quantified by estimating model error, which can be compared with the sampling error associated with the sampling of students. The PISA 2009 dataset for the cognitive domains mathematics, reading, and science is used as an example of the choice of the IRT model. It turned out that the three-parameter logistic IRT model with residual heterogeneity and a three-parameter IRT model with a quadratic effect of the ability θ provided the best model fit. Furthermore, model uncertainty was relatively small compared to sampling error regarding country means in most cases but was substantial for country standard deviations and percentiles. Consequently, it can be argued that model error should be included in the statistical inference of educational large-scale assessment studies.

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

“…The 3PLRH model has been successfully applied to LSA data and often resulted in superior model fit compared to the three-parameter logistic model (3PL; [ 48 ]) that includes a guessing parameter instead of an asymmetry parameter [ 51 , 52 , 53 , 54 ]. In this study, we did not include the 3PL model for two reasons, even though the PISA test includes multiple-choice items.…”

“…Alternative models might be preferable if the goal is to adjust for guessing effects adequately [ 55 , 57 ]. In a previous study, we demonstrated that the 3PL model did not substantially improve the model fit compared to the 2PL model [ 54 ]. In contrast, the 3PLRH model significantly improved the model fit in terms of information criteria [ 54 ].…”

“…In a previous study, we demonstrated that the 3PL model did not substantially improve the model fit compared to the 2PL model [ 54 ]. In contrast, the 3PLRH model significantly improved the model fit in terms of information criteria [ 54 ]. The 3PLRH model is able to account for guessing and slipping effects, as well as for asymmetry in item response functions [ 53 ].…”

“…To our knowledge, the 3PLRH has not yet been implemented in the operational practice of any important educational LSA study. The choice of the IRT model in LSA studies has been investigated in [ 54 , 58 , 59 , 60 , 61 ].…”

“…Model uncertainty is quantified as the model error (ME) that is computed as the square root of average squared parameter deviations (see [ 12 , 54 ]) …”