An explanatory mixture IRT model for careless and insufficient effort responding in self‐report measures

Abstract

Careless and insufficient effort responding (C/IER) on self‐report measures results in responses that do not reflect the trait to be measured, thereby posing a major threat to the quality of survey data. Reliable approaches for detecting C/IER aid in increasing the validity of inferences being made from survey data. First, once detected, C/IER can be taken into account in data analysis. Second, approaches for detecting C/IER support a better understanding of its occurrence, which facilitates designing surveys that curb the prevalence of C/IER. Previous approaches for detecting C/IER are limited in that they identify C/IER at the aggregate respondent or scale level, thereby hindering investigations of item characteristics evoking C/IER. We propose an explanatory mixture item response theory model that supports identifying and modelling C/IER at the respondent‐by‐item level, can detect a wide array of C/IER patterns, and facilitates a deeper understanding of item characteristics associated with its occurrence. As the approach only requires raw response data, it is applicable to data from paper‐and‐pencil and online surveys. The model shows good parameter recovery and can well handle the simultaneous occurrence of multiple types of C/IER patterns in simulated data. The approach is illustrated on a publicly available Big Five inventory data set, where we found later item positions to be associated with higher C/IER probabilities. We gathered initial supporting validity evidence for the proposed approach by investigating agreement with multiple commonly employed indicators of C/IER.

Accounting for careless and insufficient effort responding in large-scale survey data—development, evaluation, and application of a screen-time-based weighting procedure

“…These limitations are overcome in recently developed latent-response mixture modeling approaches Ulitzsch et al, 2022). These allow for C/IER to vary at the screen-by-respondent or item-byrespondent level and avoid assumptions on specific C/IE response patterns.…”

“…These allow for C/IER to vary at the screen-by-respondent or item-byrespondent level and avoid assumptions on specific C/IE response patterns. To facilitate distinguishing between C/IE and attentive responses, latent-response models leverage collateral item-level information, either drawing on response times from computerized questionnaires or using information on item features such as position or text length (Ulitzsch et al, 2022). For attentive responses and, if considered, response times, customary models for polytomous data (e.g., Muraki, 1997) and response times from non-cognitive assessments (Ferrando & Lorenzo-Seva, 2007;Molenaar, Tuerlinckx, Maas, & van der Maas, 2015) are employed.…”

“…More sophisticated weighting and model-based approaches overcome these limitations. However, these approaches are currently either capable of detecting only some specific types of C/IE response patterns , or are computationally too demanding to pose feasible alternatives in the context of large-scale data sets Ulitzsch, Yildirim-Erbasli, Gorgun, & Bulut, 2022).…”

Model-agnostic unsupervised detection of bots in a Likert-type questionnaire

“…A separate tradition (Hong & Cheng, 2018;Jin, Chen, & Wang, 2018;Roman, Brandt, & Miller, 2022;Ulitzsch, Yildirim-Erbasli, Gorgun, & Bulut, 2022) goes in the opposite direction, relying on measurement models and being unsupervised, which is without labeled data. Conceptually, methods therein estimate measurement model parameters iteratively in such a way that progressively downweights suspicious respondents.…”

“…In Roman et al (2022), the contaminated data are envisaged as a mixture of two CFA models whose parameters are then estimated by Markov chain Monte Carlo (MCMC). Optionally, even without class labels, NRIs may be instrumentally used to augment estimation of measurement model parameters (Hong & Cheng, 2018;Roman et al, 2022;Ulitzsch et al, 2022). On one hand, not needing a training set is an obvious advantage.…”