Battling bots: Experiences and strategies to mitigate fraudulent responses in online surveys

Abstract

Declining survey response rates have driven many researchers to seek out cost-effective methods of increasing participation, such as conducting surveys online, paying incentives, and using social media to engage hard-to-reach populations. Malicious actors can exploit the monetary incentives and anonymity of online surveys, threatening the integrity of survey data.We share two recent experiences conducting online surveys that were inundated with fraudulent responses. Our objective is to increase awareness of this emerging issue and offer guidance for others to mitigate the effects of fraudulent responders in their own research.

Estimating the benefits to Florida households from avoiding another Gulf oil spill using the contingent valuation method: Internal validity tests with probability‐based and opt‐in samples

“…Our findings are of growing relevance as opt-in samples have been increasingly used due to their low cost (Goodrich et al, 2023;Johnston, 2021). Given their low cost and accessibility to researchers without large budgets, it is likely that opt-in data will continue to be used.…”

“…In many cases, opt‐in samples may likely be the least expensive of all survey modes. One drawback of data from an opt‐in sample is that respondents may pay little attention to the details of the valuation questions, resulting in relatively low‐quality data (e.g., Giguere et al, 2020; Goodrich et al, 2023). Johnston et al (2017) assert that the highest quality surveys use probability‐based sampling and the Dillman repeat contacts method for quality assurance.…”

Comparing water quality valuation across probability and non‐probability samples

“…Methods to reduce or detect satisficing and fraudulent behavior include attention checks, bogus questions, open‐ended questions, self‐reports of effort, response times, analyzing choice response patterns and selection of non‐substantive responses, digital fingerprinting, as well as many other methods (Aguinis et al, 2021; Baker et al, 2010; Brazhkin, 2020; Chmielewski & Kucker, 2020; Lawler et al, 2021; Teitcher et al, 2015). 2 One recent study (Goodrich et al, 2023) details experiences of conducting online surveys targeted towards narrow populations (e.g., posting to social media site) that received a high number of fraudulent responses. To identify fraudulent responses, the researchers used respondent statistics, institutional knowledge, and inconsistencies, and the researchers suggest steps to prevent fraud such as sharing on social media with caution and not automating participant incentive payments.…”

Hypothetical bias mitigation in representative and convenience samples

“…As well, probability sampling, aside from simple random sampling, may require substantial expertise and experience for successful implementation. Finally, changes in modes of communication and behavior have introduced or exacerbated some of these challenges, including low coverage (Brick et al, 2006) (e.g., populations with landlines, cellphones, smartphones, and/or internet access), falling participation (Czajka & Beyler, 2016), potential non-response bias (Groves, 2006), and even fraudulent responses stemming from online "bots" (Goodrich et al, 2023).…”

“…Non‐probability or convenience samples often suffer from coverage bias and fail to represent the population of interest. Such concern continues to grow as the incidence of fraudulent responses in online convenience‐based sampling (Goodrich et al, 2023). Modern comparisons continue to showcase the non‐equivalence of non‐probability online samples to probability‐based samples (Yeager et al, 2011).…”