When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Abstract

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms.

A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item‐level non‐response

“…Various studies have related the occurrence of item omissions to lack of examinee motivation (Cosgrove, 2011; Jakwerth & Stancavage, 2003; Köhler, Pohl, & Carstensen, 2015a; Verbić & Tomić, 2009; Wise & Gao, 2017). Decline in test scores over time, for instance, has been attributed to a decline in examinee motivation, with an increase in omission rates taken as an indicator of examinee disengagement (Cosgrove, 2011; Sachse, Mahler, & Pohl, 2019). Likewise, it has been suggested to employ the rate of item omissions on background questionnaires as an indicator of disengagement in cognitive assessments, with the rationale being that examinees who are not motivated to fill out the background questionnaire might also be less motivated to engage with the items of the cognitive assessment (Boe et al , 2002).…”

Performance of missing data approaches under nonignorable missing data conditions

“…The simulated condition suffices for evaluating for which missing data mechanisms the different approaches are suited. For more information on the impact of different amounts of missing values or a different correlation between ability and missing propensity, we refer the reader to previous work (Holman & Glas, 2005;Rose, von Davier, & Xu, 2010;Sachse, Mahler, & Pohl, 2019). Fourth, we did not make use of all information available in current testing.…”

Reflections on Analytical Choices in the Scaling Model for Test Scores in International Large-Scale Assessment Studies

“…It is important to emphasize that the adjustments-and hence the scoring rules for ability-in the SA+O model will differ from country to country because the relationship of ability, response propensity, and speed differ across countries (Sachse et al, 2019;Pohl et al, 2021). In our view, a country comparison that does not employ the same scoring rule for each country cannot be considered characterized as valid (or fair).…”