The Generalizability of Online Experiments Conducted During The COVID-19 Pandemic

Abstract

The disruptions of the COVID-19 pandemic led many social scientists toward online survey experimentation for empirical research. Generalizing from the experiments conducted during a period of persistent crisis may be challenging due to changes in who participates in online survey research and how the participants respond to treatments. We investigate the generalizability of COVID-era survey experiments with 33 replications of 12 pre-pandemic designs fielded across 13 surveys on American survey respondents obtained from Lucid between March and July of 2020. We find strong evidence that these experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes that are possibly explained by increased inattentiveness. These findings mitigate concerns about the generalizability of online research during this period. The pandemic does not appear to have fundamentally changed how subjects respond to treatments, provided they pay attention to treatments and outcome questions. In this light, we offer some suggestions for renewed care in the design, analysis, and interpretation of experiments conducted during the pandemic.

Association of Acute Symptoms of COVID-19 and Symptoms of Depression in Adults

“…Moreover, as a web-based survey, we cannot estimate a response rate as with more traditional survey designs; however, we note that surveys using similar methods have demonstrated replicable results during COVID-19. 5 As respondents did not see the survey topic until entering the survey itself, it is unlikely our results are enriched for those with particular interest in, or impact from, COVID-19.…”

Mass support for proposals to reshape policing depends on the implications for crime and safety

“…Arechar and Rand (2020) and Aronow et al (2020) provide evidence of rising inattention among survey takers sourced from Amazon Mechanical Turk (MTurk) and Lucid throughout 2020. Peyton et al (2020) formalize the conditions under which this inattention uniformly shrinks treatment effect estimates toward zero and, using a series of replication studies fielded in 2020, show that estimates among attentive respondents are closer to previously published estimates. They recommend that researchers conducting online surveys in 2020 -particularly via the Lucid platform -should use attention checks placed near the beginning of the survey to screen out participants that provide low-quality data (e.g., speeding through surveys without reading question content).…”

“…Given recent declines in attentiveness among online survey respondents we removed inattentive respondents at the beginning of the survey (i.e., pre-treatment; Aronow et al 2020;Peyton et al 2020; see Online Appendix C.1). Additionally, we constructed survey weights to adjust for differences in respondent demographics using target proportions from the American Community Survey (see Online Appendix C.2).…”

“…(Coppock and McClellan, 2018). Although descriptive estimates of public opinion obtained from quota samples may differ significantly from nationally representative opinion polls, this is not generally a concern for survey experimental estimates (Mullinix et al, 2015;Coppock et al, 2018;Coppock, 2019;Coppock and McClellan, 2019;Peyton et al, 2020).…”

“…A third possibility is that these movements are perceived as a threat to the traditional roles that police play in crime control. Most people believe police should continue to play the primary role in responding to and investigating crimes (Peyton et al 2020), and are opposed to any reductions in spending on police services (Pew 2020). Given that Americans perceive crime levels to be higher than they actually are (e.g., Esberg and Mummolo 2018) and believe police are a cost-effective source of crime prevention (Metcalfe and Pickett, 2018), opposition to abolition and defunding may arise because people think the changes these movements seek will reduce police departments' abilities to address crime and disorder.…”