Response time-based treatment of omitted responses in computer-based testing

Abstract

A new response time-based method for coding omitted item responses in computer-based testing is introduced and illustrated with empirical data. The new method is derived from the theory of missing data problems of Rubin and colleagues and embedded in an item response theory framework. Its basic idea is using item response times to statistically test for each individual item whether omitted responses are missing completely at random (MCAR) or missing due to a lack of ability and thus not at random (MNAR) with fixed type-1 and type-2 error levels. If the MCAR hypothesis is maintained, omitted responses are coded as not administered (NA), and as incorrect (0) otherwise. The empirical illustration draws from the responses given by = 766 students to 70 items of a computer-based ICT-skills test. The new method is compared with the two common deterministic methods of scoring omitted responses as 0 or as NA. In result, response time thresholds from 18 to 58 seconds were identified. With 61 %, more omitted responses were recoded into 0 than into NA (39 %). The differences in difficulty were larger when the new method was compared to deterministically scoring omitted responses as NA compared to scoring omitted responses as 0. The variances and reliabilities obtained under the three methods showed small differences. The paper concludes with a discussion of the practical relevance of the observed effect sizes, and with recommendations for the practical use of the new method as a method to be applied in the early stage of data processing.

A Response Time Process Model for Not‐Reached and Omitted Items

“…Two assumptions that are made in the new model are: (1) the omission happens when the perceived effort, hence the expected RT to answer the item, is higher than the amount of time a person allocates to the item. This assumption was inferred from prior studies (Frey et al., 2018; Lee & Ying, 2015; Weeks et al., 2016; Wolf et al., 1995) and was further verified in our real data example; (2) we assume that the expected RTs, , and the censoring RTs, , are independent, which is one of the most common assumptions made in survival analysis (Lee & Ying, 2015). Apparently, the independent censoring assumption greatly simplifies our model estimation.…”

A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item‐level non‐response

“…The SA+E framework complements and refines recent approaches for examinee disengagement as well as non‐ignorable item omissions. Compared to RT‐based scoring methods separating engaged and disengaged responses and/or item omissions by defining RT thresholds (Frey et al , 2018; Lee & Jia, 2014; Wise & DeMars, 2006), the SA+E framework comes with less strict assumptions concerning RT distributions associated with engaged and disengaged behaviour since these are allowed to overlap. Compared to previous model‐based approaches for identifying disengaged examinee behaviour (Meyer, 2010; Pokropek, 2016; Schnipke & Scrams, 1997; Wang & Xu, 2015), the model allows disengaged behaviour to vary across both items and examinees while considering engagement when estimating ability.…”

“…Otherwise, item omissions are considered not attempted and treated as missing responses in all further analyses (Yamamoto, Khorramdel, & von Davier, 2013). Recent approaches for RT‐based scoring of omitted responses extend this rationale by allowing for item‐specific, empirically derived thresholds (Frey, Spoden, Goldhammer, & Wenzel, 2018; Weeks, von Davier, & Yamamoto, 2016).…”

About Still Nonignorable Consequences of (Partially) Ignoring Missing Item Responses in Large-scale Assessment

“…As an alternative to ignoring missing item responses, multidimensional IRT models for nonignorable item responses have been proposed (Debeer et al 2017;Frey et al 2018;Fu et al 2010;Glas et al 2015;Holman and Glas 2005;Köhler et al 2015a;Kreitchmann et al 2018;Kuha et al 2018;Lu and Wang 2020;Okumura 2014;Pohl and Becker 2020;Rosas and Shomer 2008;Rose et al 2015Rose et al 2017Rose et al 2010Santos et al 2016;Ulitzsch et al 2020;Weeks et al 2016;Zhou and Huggins-Manley 2019). The two-dimensional model of Holman and Glas (2005) introduces a latent individual response propensity (response tendency) ξ (besides the latent ability θ), which predicts missingness in item responses.…”