Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Abstract

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents’ true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding. Expected final online publication date for the Annual Review of Psychology, Volume 74 is January 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Open Science Perspectives on Machine Learning for the Identification of Careless Responding: A New Hope or Phantom Menace?

“…Numerous definitions of careless responding exist in the literature, with likely the most commonly used definition being "a response set in which the respondent answers a survey measure with low or little motivation to comply with survey instructions, correctly interpret item content, and provide accurate responses" (Huang et al, 2012). An alternative definition is given by Ward & Meade (2022): "careless responding occurs when participants are not basing their response on the item content, and it can occur when a respondent does not read an item, does not understand an item, or is unmotivated to think about what the item is asking." The latter definition highlights that careless responding may not be intentional (e.g., Huang et al, 2012;Ward & Pond, 2015;Ward et al, 2017), for instance when a participant misunderstands an item due to poor or ambiguous wording.…”

“…These types of response bias are characterized by a respondent's intention to systematically misrepresent their true score for a certain scale-which requires careful content-dependent responding-while careless respondents do not have this intention since their responses are content-independent (cf. Ward & Meade, 2022).…”

“…In the most extreme case, this boils down to straightlining, i.e., always giving the same response (also known as longstring; Johnson, 2005). Inconsistent careless responses "do not match patterns that would be expected based on theoretical/logical grounds or trends in the data" (Ward & Meade, 2022). Hence, such responses fail to meet an expected level of consistency.…”

“…In other words, inconsistent responding is characterized by choosing answer categories at random, for instance near the scale endpoints (extreme responding; Bachman & O'Malley, 1984) or with equal probability across all categories. Fast responses are responses that have been given at such a speed that renders it arguably impossible that a respondent has "read, understood, and responded accurately to the survey items" (Ward & Meade, 2022). Indeed, excessively fast responses are indicative of carelessness (e.g., Bowling et al, 2021bBowling et al, , 2016Huang et al, 2012;Meade & Craig, 2012).…”

“…This phenomenon is known as careless responding in the psychology literature and has been identified as a major threat to the validity of research results (e.g., Arias et al, 2020;Huang et al, 2015b;McGrath et al, 2010;Meade & Craig, 2012;Woods, 2006). Careless responding is widely prevalent and suspected to be present in all survey data (Ward & Meade, 2022). It is broadly recommended to screen and filter survey data for careless responses (e.g., Arias et al, 2020;Huang et al, 2015b;Meade & Craig, 2012), for which a plethora of different methods exist (see, e.g., Table 1 in Arthur et al, 2021, for an overview).…”