A Response-Time-Based Latent Response Mixture Model for Identifying and Modeling Careless and Insufficient Effort Responding in Survey Data

Abstract

Careless and insufficient effort responding (C/IER) can pose a major threat to data quality and, as such, to validity of inferences drawn from questionnaire data. A rich body of methods aiming at its detection has been developed. Most of these methods can detect only specific types of C/IER patterns. However, typically different types of C/IER patterns occur within one data set and need to be accounted for. We present a model-based approach for detecting manifold manifestations of C/IER at once. This is achieved by leveraging response time (RT) information available from computer-administered questionnaires and integrating theoretical considerations on C/IER with recent psychometric modeling approaches. The approach a) takes the specifics of attentive response behavior on questionnaires into account by incorporating the distance–difficulty hypothesis, b) allows for attentiveness to vary on the screen-by-respondent level, c) allows for respondents with different trait and speed levels to differ in their attentiveness, and d) at once deals with various response patterns arising from C/IER. The approach makes use of item-level RTs. An adapted version for aggregated RTs is presented that supports screening for C/IER behavior on the respondent level. Parameter recovery is investigated in a simulation study. The approach is illustrated in an empirical example, comparing different RT measures and contrasting the proposed model-based procedure against indicator-based multiple-hurdle approaches.

An explanatory mixture IRT model for careless and insufficient effort responding in self‐report measures

“…However, Ulitzsch et al. (2021a) illustrated that multiple‐hurdle approaches are also heavily impacted by threshold choices.…”

“…Ulitzsch et al. (2021a) presented a model‐based approach for online‐administered surveys that leverages response time data to identify and model C/IER at the screen‐by‐respondent level. The approach originates in model‐based approaches that jointly consider responses and response times for modelling non‐effortful behaviour in cognitive assessments (Liu et al., 2020; Meyer, 2010; Nagy & Ulitzsch, 2021; Schnipke & Scrams, 1997; Ulitzsch et al., 2020; Wang & Xu, 2015) and has been adapted to the specifics of response behaviour on self‐report measures.…”

“…In a simulation study, Ulitzsch et al. (2021a) showed that the approach can deal with manifold types of C/IER patterns within the same data set, given that the model's assumptions hold and C/IE responses are unrelated to the trait to be measured. The applicability of the approach, however, is limited as it requires access to response time information on the item level.…”

“…A potential reason for the scarcity of studies investigating item characteristics related to the occurrence of C/IER may be the lack of tools available, as the vast majority of current approaches identifies C/IER at the respondent level (most indicator‐based approaches as well as the model‐based approaches by Arias et al., 2020; Meade & Craig, 2012; Schroeders et al., 2020) or screen level (Galesic & Bosnjak, 2009; Ulitzsch et al., 2021a). The aim of the present study is to overcome this limitation and equip researchers with an approach that facilitates the investigation of item‐level characteristics associated with C/IER.…”

“…Schematic illustration of different careless and insufficient effort response patterns, adapted from Ulitzsch, Pohl, Khorramdel, Kroehne, and von Davier (2021a). …”