Fraud in Online Surveys: Evidence from a Nonprobability, Subpopulation Sample

Abstract

We hired a well-known market research firm whose surveys have been published in leading political science journals, including JEPS. Based on a set of rigorous “screeners,” we detected what appears to be exceedingly high rates of identity falsification: over 81 percent of respondents seemed to misrepresent their credentials to gain access to the survey and earn compensation. Similarly high rates of presumptive character falsification were present in panels from multiple sub-vendors procured by the firm. Moreover, we found additional, serious irregularities embedded in the data, including evidence of respondents using deliberate strategies to detect and circumvent one of our screeners, as well as pervasive, observable patterns reflecting that the survey had been taken repeatedly by a respondent or collection of respondents. This evidence offers reasons to be concerned about the quality of online nonprobability, subpopulation samples, and calls for further, systematic research.

Guns and Democracy: Anti-System Attitudes, Protest, and Support for Violence Among Pandemic Gun-Buyers

“…To minimize topical selection bias, we did not inform respondents of the purpose of the survey when they entered it, and questions covered a broad range of topics, mostly related to public health (the questions regarding guns also came late in the survey, making self-selection based on guns very unlikely). We filtered out inattentive, repeat, and semi-automated respondents through multiple closed-and open-ended attention checks to address growing concerns about fraud and inattention in online panels (Bell and Gift 2022). Emerging evidence suggests this general approach to data collection can perform as well as traditional probability sampling (Enns and Rothschild 2021;Lehdonvirta et al 2021).…”

Fraud in Online Surveys: Evidence from a Nonprobability, Subpopulation Sample – ADDENDUM

“…In the original publication of Bell and Gift (2022) a non-attribution statement for author Andrew Bell was inadvertently excluded. The non-attribution statement for Bell reads as follows:…”