The Impact of Insufficient Effort Responding Detection Methods on Substantive Responses: Results from an Experiment Testing Parameter Invariance

Abstract

Researchers using survey methods can choose among a variety of methods aimed at detecting insufficient effort responding among the participants in their studies. Some of these methods entail modifying the survey questionnaire by adding bogus items, instructed-response items, or instructed manipulation checks. While these methods have been found effective in detecting insufficient effort responding, it remains unclear whether their presence in a questionnaire can affect responses to items of substantive research interest. We conducted an experiment investigating this potential impact in a sample (N 5 1,092) of working adults. Adopting an invariance testing approach, we assessed whether employing bogus items, instructed-response items, or instructed manipulation checks, with or without warning study participants, respectively, would lead to non-invariant estimates of parameters for substantive variables. Results suggest that, while most parameter estimates were invariant to the use of insufficient effort responding detection methods, the reliability of measurements may be adversely impacted, posing a threat to construct validity and statistical conclusion validity. However, reliability might be maintained when participants are warned at the beginning of the questionnaire. Results also suggest that bogus items may have some advantages over other methods in terms of parameter invariance. We discuss the implications of our findings.

Methods of Detecting Insufficient Effort Responding: Comparisons and Practical Recommendations

“…Responses not following the instruction or deemed unreasonable by common sense (e.g., never brushed one’s teeth) suggest IER. Unfortunately, such methods require changing the survey or questionnaire (e.g., by adding bogus or instructed-response items), which may have unintended effects on response behavior (Breitsohl & Steidelmüller, 2018). Other ancillary information such as response times may also be used to identify speeded responders, wherein individuals who respond too quickly are flagged for IER (Marianti, Fox, Avetisyan, Veldkamp, & Tijmstra, 2014).…”

Insufficient Effort Responding in Experimental Philosophy

“…Second, attention checks' required responses should vary within studies (e.g., not always "strongly disagree", or the farthest left response), so that they are more likely to detect IER subjects who continuously provide the same response (Desimone et al 2015;Huang et al 2014;Meade and Craig 2012). Third, there must not be plausible alternative readings of the attention checks that justify differing responses (such as, e.g., with the purported bogus sentence "I work twenty-eight hours in a typical work day" which might be read as expressing, in a hyperbolic way, that one is hardworking, see Breitsohl and Steidelmüller 2018). And fourth, all checks should be formulated in ways that minimize unwanted effects, such as emotions, amusement or answers based on social desirability (for this problem see again the above bogus sentence about a twenty-eight hours work day).…”

Scrutinizing Social Identity Theory in Corporate Social Responsibility: An Experimental Investigation

“…Fourteen participants were excluded based on a speed index ( Leiner, 2019 ). They completed the survey three times as fast (under 4 min 20 s) as the average respondent did ( Mdn = 14 min 27 s), so they could not have been able to read the vignette and items carefully ( Breitsohl and Steidelmüller, 2018 ).…”