Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships

Abstract

Competence data from low‐stakes educational large‐scale assessment studies allow for evaluating relationships between competencies and other variables. The impact of item‐level nonresponse has not been investigated with regard to statistics that determine the size of these relationships (e.g., correlations, regression coefficients). Classical approaches such as ignoring missing values or treating them as incorrect are currently applied in many large‐scale studies, while recent model‐based approaches that can account for nonignorable nonresponse have been developed. Estimates of item and person parameters have been demonstrated to be biased for classical approaches when missing data are missing not at random (MNAR). In our study, we focus on parameter estimates of the structural model (i.e., the true regression coefficient when regressing competence on an explanatory variable), simulating data according to various missing data mechanisms. We found that model‐based approaches and ignoring missing values performed well in retrieving regression coefficients even when we induced missing data that were MNAR. Treating missing values as incorrect responses can lead to substantial bias. We demonstrate the validity of our approach empirically and discuss the relevance of our results.

A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item‐level non‐response

“…While scoring item omissions as wrong assumes the probability of a correct response to an omitted item to be zero (Rose, von Davier, & Xu, 2010), ignoring item omissions implies ignorability (Rose et al , 2010). In the case that ignorability does not hold, ignoring missing data jeopardizes validity of inference and can induce bias to person and item parameter estimates (de Ayala, Plake, & Impara, 2001; Culbertson, 2011; Finch, 2008; Köhler, Pohl, & Carstensen, 2015b, 2017; Pohl et al , 2014; Rose, 2013; Rose et al , 2010).…”

A Mixture IRTree Model for Performance Decline and Nonignorable Missing Data

“…The literature has proposed at least three approaches for handling nonignorable missing data: substituting incorrect answers, ignoring nonresponses, and using model-based approaches. The first two approaches depend on specific assumptions that may not be met in real testing situations and are more likely to result in biased estimation and incorrect inferences (Holman & Glas, 2005; Köhler et al, 2017; Pohl et al, 2014). Furthermore, the processes underlying skipped items and not-reached items should be differentiated; therefore, the model-based approaches are thought of as a better way to deal with nonresponses than previous methods.…”

Performance of missing data approaches under nonignorable missing data conditions

“…When the missing mechanism is MCAR, MAR, or nonignorable as modeled by the approach for nonignorable missing responses, incorrect scoring results in an underestimation of ability for persons with missing values (e.g., Culbertson, 2011;De Ayala, Plake, & Impara, 2001;Finch, 2008;Lord, 1974;Rose et al, 2010). Ignoring missing values performs well in cases where the missing mechanism is MCAR or MAR (Culbertson, 2011;Finch, 2008;Köhler et al, 2017;Rose et al, 2010) 2 . Ignoring results in biased parameter estimates when the missing mechanism is nonignorable (Culbertson, 2011;De Ayala, Plake, & Impara, 2001;Pohl, Gräfe, & Rose, 2014;Rose et al, 2010).…”

“…Ignoring results in biased parameter estimates when the missing mechanism is nonignorable (Culbertson, 2011;De Ayala, Plake, & Impara, 2001;Pohl, Gräfe, & Rose, 2014;Rose et al, 2010). If the nonignorable missing mechanism follows the model for nonignorable missing values, the bias of ignoring missing values becomes negligible when the relationship between ability and missing propensity is rather small (Holman & Glas, 2005;Köhler et al, 2017;Pohl et al, 2014). The approach for nonignorable missing values performs well when the missing mechanism is MCAR, MAR, or when the missing mechanism is generated according to the model for nonignorable missing values (Holman & Glas, 2005;Rose, 2013;Rose et al, 2010).…”

“…The approach for nonignorable missing values has not at all been evaluated on this missing mechanism. Those studies evaluating the performance of the approach for nonignorable missing data exclusively used the mechanism described in Equation 2 (e.g., Glas et al, 2015;Holman & Glas, 2005;Köhler et al, 2017;Pohl et al, 2014;Rose et al, 2010).…”