Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys

Abstract

WARD, MARY KATHRINE. Using Virtual Presence and Survey Instructions to Minimize Careless Responding on Internet-Based Surveys. (Under the direction of Dr. Samuel B. Pond.) Internet-based survey data inform knowledge creation in research and justify work activities in many organizations. While there are decided advantages to Internet-based surveys, this mode of administration comes with its own set of challenges. Survey respondents may intentionally or unintentionally respond to the survey in a manner that does not accurately reflect their true beliefs or feelings. The purposes of this study were to examine two approaches that address the problem of careless responding (CR), and increase attentiveness among respondents with better survey design. This study investigated instructional manipulation and virtual human presence as potential buffers against CR. The sample consisted of undergraduate students who voluntarily completed an Internet-based survey. This study used a 3x3 between-subjects experimental design where virtual presence (absent, animated shape, and virtual human) and type of instruction (anonymous, warning, and feedback) were the independent variables. Indicators of CR were the dependent variables. Findings showed that warning instructions significantly reduced some forms of CR, but promising feedback had little effect on CR. The interaction of instructions and virtual presence had a significant effect on CR. Future research will need to tease apart the nuances of the relationship between instructions, virtual presence, and CR. The discussion includes implications for Internet-based survey administration and future directions for addressing the problem of CR.

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

“…Data on person-level confounding variables (eg, reading speed and cognitive skills) were unavailable; adjusting for these variables, as well as item-level variables (eg, item complexity and length), could also improve the signal-to-noise ratio of the data through a richer statistical model 28,52–56,58–61. Nevertheless, the direct use of the total amount of time taken on an assessment (ie, the raw completion time) had been found to have utility in previous work,12,37,62–65 including the identification of aberrant answering 8,21,57…”

“…In the context of educational testing, van der Linden and van Krimpen-Stoop noted that an individual’s response times are distinct from his/her answers; therefore, the response times may provide additional information about an individual’s level of aberrant responding that cannot be gleaned from the answer choices themselves 11. In a general assessment context, unduly fast responses may signify aberrant response behavior, for example, that an individual is responding carelessly in an effort to complete the assessment as quickly as possible, or is mechanically “faking good” in his/her responses, without deep consideration of the most accurate answer, when it is a simple matter to do so 8,13–21. On the other hand, unduly slow responses may also signify aberrant response behavior, for example, an individual who takes an unduly long time to provide responses may be doing so because he/she is thoroughly assessing the answer choices to determine which is the most socially desirable, when making such a determination is not trivial 22–24.…”

Are Attention Check Questions a Threat to Scale Validity?

“…Research has shown that giving warnings can effectively reduce careless responses. Explicit warning instructions such as “…responding without much effort will be flagged for low‐quality data” (Ward & Pond, ) and “…responding without much effort would result in loss of credits” (Huang et al, ) were shown to increase quality in responses. Because survey respondents who read the attention check can tell it is a “trap”, it signifies that the researchers are flagging low‐quality data, a function that resembles what a warning basically does.…”

“…Researchers' use of attention checks may vary slightly, but they converge on two major forms. Perhaps the most popular form of attention check is instructed‐response items (see Bowling et al, ; Meade & Craig, ; Ward & Pond, ), which are items embedded in a scale with an obvious correct answer. The earlier example—“please select four for this item” and “please select moderately inaccurate for this item” (Huang, Curran, Keeney, Poposki, & DeShon, )—are common instructed‐response items.…”

Applying Social Psychology to Prevent Careless Responding during Online Surveys

“…Bowling, Huang, Bragg, Khazon, Liu, & Blackmore, ). In three studies, we use social psychological theories to develop and test three prevention strategies (Ward & Pond III, ) related to increasing respondent motivation to respond carefully. Study 1 presented control, scripted, or video‐recorded instructions designed to increase the social influence of survey administrators on survey participants.…”