Applying Social Psychology to Prevent Careless Responding during Online Surveys

Abstract

related to increasing respondent motivation to respond carefully. Study 1 presented control, scripted, or video-recorded instructions designed to increase the social influence of survey administrators on survey participants. Participants in the control group were significantly more likely to admit to CR than the script and video groups. Compared with the control, scripted instructions decreased interest, and had no effect on objective indicators of CR. Study 2 found that instructions designed to induce cognitive dissonance increased logical consistency of responses and survey interest. Instructions to create a sense of hypocrisy increased accuracy on instructedresponse items. Study 3 showed that leveraging social exchange theory in survey instructions generally had no effect on CR. Similar results were found for both continuous and dichotomous scoring of indicators of CR across the three studies. Results demonstrate that facets of CR can be influenced via survey design. Future studies are needed to develop a more thorough understanding of best practices in survey design with respect to preventing CR.

Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices

“…To calculate an odd-even consistency index, each scale of a test is divided into odd and even items, and scores are computed for each half. The correlation between the resulting scores across multiple scales is used as a unidimensional index of response consistency (Curran, 2016;Johnson, 2005;Meade & Craig, 2012;Ward & Meade, 2018). Consistent with the other correlation indices, the Spearman-Brown correction formula is used and the values are bound between -1 and 1.…”

“…& Consistency indices Consistency indices rely on the assumption that Bsufficient effort^responding differs from random responding in terms of stability of response patterns and coherence regarding items related to the same dimensions. Five consistency indices exist based on psychometric antonyms, psychometric synonyms, odd-even consistency (Curran, 2016;Meade & Craig, 2012;Niessen et al, 2016;Ward & Meade, 2018), and a multidimensional index of response reliability resulting from Gendre's functional method (Dupuis, Meier, Capel, & Gendre, 2015). The fifth index, response coherence, is another that Gendre developed to measure the part of variance in individual responses explained by the factor structure of the questionnaire.…”

“…& Outlier statistics As their name suggests, outlier statistics are used to detect response sets that are particularly uncommon in comparison to the rest of a sample. They include two measures: Mahalanobis distance (Curran, 2016;Niessen et al, 2016;Ward & Meade, 2018) and persontotal correlation (Curran, 2016). The Mahalanobis distance is used to detect multivariate outliers using a chi-square test to determine significantly different variable sets, and assumes they are outlying.…”

“…& Long-string analysis Long-string analysis consists of calculating the number of consecutive identical responses throughout a questionnaire. This index is intuitive and frequently used to detect insufficient-effort responding patterns (Curran, 2016;Johnson, 2005;Meade & Craig, 2012;Ward & Meade, 2018). However, it is not useful for identifying strict random responding.…”

Social Interaction and Internet‐Based Surveys: Examining the Effects of Virtual and In‐Person Proctors on Careless Response

“…Thus, low response times indicate more carelessness. Researchers tend to compute this indicator using survey page response time and examining response time per item, with a threshold of anything below 2 seconds per item, for example, indicating carelessness (e.g., Bowling et al, ; Huang et al, ; Ward & Meade, ). The survey software utilised for this study did not automatically collect page response time, so we utilised total survey response time (the difference between survey start time and survey end time) to calculate the response time CR indicator.…”