Accounting for careless and insufficient effort responding in large-scale survey data—development, evaluation, and application of a screen-time-based weighting procedure

Ulitzsch, Esther; Shin, Hyung-Ki; Lüdtke, Oliver

doi:10.3758/s13428-022-02053-6

Cited by 14 publications

(22 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The probabilistic filtering approach can be understood as a special case of weighting‐based adjustment procedures to NRB (Hong & Cheng, 2019; Mislevy & Bock, 1982; Schuster & Yuan, 2011; Ulitzsch et al., 2023). By allowing to adjust for NRB on the item‐by‐person level in the estimation of both item and person estimates, however, the probabilistic filtering approach overcomes a key limitation of weighting‐based adjustment procedures, which either only support person‐level adjustments (Mislevy & Bock, 1982; Ulitzsch et al., 2023) or assume item parameters to be known (Mislevy & Bock, 1982; Schuster & Yuan, 2011). Note that this comes at the price of a slight increase in computational burden since multiple data sets need to be analyzed.…”

Section: Discussionmentioning

confidence: 99%

“…Ulitzsch et al. (2023) developed a screen‐time‐based weighting approach designed for computerized questionnaire administrations in which times spent on screen are recorded. In Step 1, this approach draws on mixture modeling to decompose log screen time distributions into several subcomponents, out of which the subcomponent with the lowest mean is assumed to stem from NRB.…”

Section: Approaches To Identifying and Accounting For Non‐effortful R...mentioning

confidence: 99%

“…The mixture decomposition approach is based on the assumption that response times stemming from NRB result in response time distributions that differ from those associated with SB (Meyer, 2010; Ranger & Kuhn, 2017; Ulitzsch et al., 2023). Item‐level log response times form the basis for creating the filtering model of the mixture decomposition approach.…”

Section: Developing a Probabilistic Filtering Approachmentioning

confidence: 99%

“…In the case that log response times are decomposed into at least two components (i.e.,

C_j\ge 2

), the component with the lowest mean is assumed to consist of responses due to NRB (Ulitzsch et al., 2023). That is

$$\begin{equation} c^{\text{NRB}}_j = \underset{c_j\,\in \lbrace 1, \ldots , C_j\rbrace }{\text{arg} \min } \mu _{c_j}.

…”

Section: Developing a Probabilistic Filtering Approachmentioning

confidence: 99%

“…Following Ulitzsch et al. (2023), after decomposing log item response time distributions, posterior NRB class probabilities

\pi _{ij}^{\text{NRB}}=p(z_{ij}=c^{\text{NRB}}_j|\ln (t_{ij}))

are determined for each examinee. Subsequently, these are employed for creating multiple filtered response data sets by setting responses to missing with probability

\pi _{ij}^{\text{NRB}}

.…”

Section: Developing a Probabilistic Filtering Approachmentioning

confidence: 99%

See 4 more Smart Citations

A Probabilistic Filtering Approach to Non‐Effortful Responding

Ulitzsch

Domingue²,

Kapoor³

et al. 2023

Educational Measurement

View full text Add to dashboard Cite

Common response‐time‐based approaches for non‐effortful response behavior (NRB) in educational achievement tests filter responses that are associated with response times below some threshold. These approaches are, however, limited in that they require a binary decision on whether a response is classified as stemming from NRB; thus ignoring potential classification uncertainty in resulting parameter estimates. We developed a response‐time‐based probabilistic filtering procedure that overcomes this limitation. The procedure is rooted in the principles of multiple imputation. Instead of creating multiple plausible replacements of missing data, however, multiple data sets are created that represent plausible filtered response data. We propose two different approaches to filtering models, originating in different research traditions and conceptualizations of response‐time‐based identification of NRB. The first approach uses Gaussian mixture modeling to identify a response time subcomponent stemming from NRB. Plausible filtered data sets are created based on examinees' posterior probabilities of belonging to the NRB subcomponent. The second approach defines a plausible range of response time thresholds and creates plausible filtered data sets by drawing multiple response time thresholds from the defined range. We illustrate the workings of the proposed procedure as well as differences between the proposed filtering models based on both simulated data and empirical data from PISA 2018.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Approaches To Identifying and Accounting For Non‐effortful R...mentioning

confidence: 99%

Section: Developing a Probabilistic Filtering Approachmentioning

confidence: 99%

“…In the case that log response times are decomposed into at least two components (i.e.,

C_j\ge 2

), the component with the lowest mean is assumed to consist of responses due to NRB (Ulitzsch et al., 2023). That is

$$\begin{equation} c^{\text{NRB}}_j = \underset{c_j\,\in \lbrace 1, \ldots , C_j\rbrace }{\text{arg} \min } \mu _{c_j}.

…”

Section: Developing a Probabilistic Filtering Approachmentioning

confidence: 99%

“…Following Ulitzsch et al. (2023), after decomposing log item response time distributions, posterior NRB class probabilities

\pi _{ij}^{\text{NRB}}=p(z_{ij}=c^{\text{NRB}}_j|\ln (t_{ij}))

are determined for each examinee. Subsequently, these are employed for creating multiple filtered response data sets by setting responses to missing with probability