On the Treatment of Missing Item Responses in Educational Large-scale Assessment Data: The Case of PISA 2018 Mathematics

Abstract

Missing item responses are prevalent in educational large-scale assessment studies like the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians advocated a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. This article uses the PISA 2018 mathematics dataset to investigate the consequences of different missing data treatments on country means. Obtained country means can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

Developing mathematics problems using local wisdom context of Maluku to improve students’ numeracy

“…Some studies show that the problem item that has been tried and validated first is better than the problem item that is not validated first (Alavi et al, 2020;Erfan et al, 2020;Lu et al, 2021). Invalid items are negligible and unfit for use (Petersen et al, 2020;Robitzsch, 2021). In addition to item validation, the mathematics problems website has gone through an expert validation process for the website, including content, language, and media.…”

Anonymiced Shareable Data: Using mice to Create and Analyze Multiply Imputed Synthetic Datasets

“…Every single variable is then imputed conditionally on a model containing only the variables in the sequence located before the variable to be imputed. This approach has been implemented in the R-packages mdmb [19] for imputation of missing data and synthpop [20] for imputation of synthetic data.…”

“…Every single variable is then imputed conditionally on a model containing only the variables in the sequence located before the variable to be imputed. This approach has been implemented in the R-packages mdmb [19] for imputation of missing data and synthpop [20] for imputation of synthetic data. Fully conditional specification (FCS) has been implemented in the mice package [14,21] in R [22], which has been developed for multiple imputation to overcome problems related to nonresponse.…”