Ensuring survey research data integrity in the era of internet bots

Abstract

We used an internet-based survey platform to conduct a cross-sectional survey regarding the impact of COVID-19 on the LGBTQ + population in the United States. While this method of data collection was quick and inexpensive, the data collected required extensive cleaning due to the infiltration of bots. Based on this experience, we provide recommendations for ensuring data integrity. Recruitment conducted between May 7 and 8, 2020 resulted in an initial sample of 1251 responses. The Qualtrics survey was disseminated via social media and professional association listservs. After noticing data discrepancies, research staff developed a rigorous data cleaning protocol. A second wave of recruitment was conducted on June 11–12, 2020 using the original recruitment methods. The five-step data cleaning protocol led to the removal of 773 (61.8%) surveys from the initial dataset, resulting in a sample of 478 participants in the first wave of data collection. The protocol led to the removal of 46 (31.9%) surveys from the second two-day wave of data collection, resulting in a sample of 98 participants in the second wave of data collection. After verifying the two-day pilot process was effective at screening for bots, the survey was reopened for a third wave of data collection resulting in a total of 709 responses, which were identified as an additional 514 (72.5%) valid participants and led to the removal of an additional 194 (27.4%) possible bots. The final analytic sample consists of 1090 participants. Although a useful and efficient research tool, especially among hard-to-reach populations, internet-based research is vulnerable to bots and mischievous responders, despite survey platforms’ built-in protections. Beyond the depletion of research funds, bot infiltration threatens data integrity and may disproportionately harm research with marginalized populations. Based on our experience, we recommend the use of strategies such as qualitative questions, duplicate demographic questions, and incentive raffles to reduce likelihood of mischievous respondents. These protections can be undertaken to ensure data integrity and facilitate research on vulnerable populations.

Sex in the Time of COVID-19: Patterns of Sexual Behavior Among LGBTQ+ Individuals in the U.S.

“…Despite these mechanisms, bots were able to evade the Qualtrics-designed bot protection questions. To address this issue, research staff developed a bot identification and data cleaning protocol which has been described in detail in a previous study (Griffin et al, 2021 ). Briefly, the survey was first launched in the first week of May 2020.…”

Employment Loss as a Result of COVID-19: a Nationwide Survey at the Onset of COVID-19 in US LGBTQ+ Populations

“…Responses flagged as bots by Google reCaptcha (Google, 2020 ) or had inconsistencies, such as duplicate responses for qualitative questions, were removed from the study sample. Further detail on bot removal has been described previously (Griffin et al, 2021 ). Eligible participants were offered a $5 electronic gift card in Wave I and the chance to win one of ten $100 gift cards in a raffle for Wave II.…”

Teacher preparation and the COVID-19 disruption: Understanding the impact and implications for novice teachers

“…While the data were screened to ensure only participants who met study criteria were included in the analysis, it is possible that non-teachers or participants who were outside of the described sample completed the survey, such as bots. Some studies have evaluated methods to deter bots from answering surveys ( Griffin et al, 2021 ; Pratt-Chapman et al, 2021 ). Measures described in the literature were utilized to reduce this risk.…”