Strategies for Detecting Insincere Respondents in Online Polling

Abstract

While the migration of public opinion surveys to online platforms has often lowered costs and enhanced timeliness, it has also created new vulnerabilities. Respondents completing the same survey multiple times from different IP addresses, overseas workers posing as Americans, and algorithms designed to complete surveys are among the threats that have emerged in this new era. This paper is an attempt to measure the prevalence of such respondents and their impact on survey data quality, while demonstrating methodological approaches for doing so. Prior studies typically examine just one platform and rely on closed-ended questions and/or paradata (e.g., IP addresses) to identify untrustworthy interviews. This is problematic because such data are relatively easy for bad actors to fake. We carried out a large-scale study with an eye toward overcoming these limitations. This study examines the threat of insincere respondents using large samples from six online platforms: three opt-in survey panels, two address-recruited survey panels, and a crowdsourced sample. Rather than relying solely on closed-ended responses, we incorporated an analysis of 375,834 open-ended answers. By their very nature, open-ended questions offer a more sensitive indicator of whether a respondent is genuine or not. The study found that the incidence of insincere respondents varied significantly by the type of online sample. Critically, insincere respondents did not just answer at random, but rather they tended to select positive answer choices, introducing a small, systematic bias into estimates like presidential approval. Two common data-quality checks failed to detect most insincere respondents.

Effect of mortality salience on charitable donations: Evidence from a national sample.

“…Aiming for a nationally representative sample, UAS members were recruited from randomly selected U.S. addresses, sampling probabilities were adjusted for underrepresented populations, and Internet-connected tablets were provided to interested individuals if needed (Alattar, Messel, & Rogofsky, 2018; USC Dornsife Center for Economic and Social Research, 2019). Address-recruited online panels tend to be better than opt-in online panels at achieving national representativeness (Tourangeau, Conrad, & Couper, 2013) and high-quality data (Kennedy et al, 2020). A sample size of 3,142 would have been sufficient to discover an effect size of at least .1 in a two-sided t test with statistical power at .8 and α at .05.…”

Using Market Research Panels for Behavioral Science: An Overview and Tutorial

“…McClelland, 2019;Mitofsky, 1989). Despite this limitation, as much as 80% of the political polls included in the respected data website FiveThirtyEight's election forecasts use nonprobability panels because such panels offer an approximation of the general population at a much lower cost than probability sampling (see Kennedy et al, 2020).…”

Explaining how long CO₂ stays in the atmosphere: Does it change attitudes toward climate change?

“…Knowledge Networks (GfK) panel members are randomly selected using an Address Based Sampling (ABS) methodology via the Delivery Sequence File (DSF) of the United States Postal Service for recruiting panel members, a technique that has been shown to reduce the rate of “bogus” responders to trace levels (Kennedy et al, 2020). Moreover, this method allows GfK to include hard-to-reach individuals, such as young adults and those from various minority groups.…”