The Generalizability of Online Experiments Conducted During the COVID-19 Pandemic

Abstract

The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.

Mass support for proposals to reshape policing depends on the implications for crime and safety

“…While it is encouraging that attitudes on key variables of interest in our survey comport well with probability surveys conducted around the same time, making descriptive inferences about the U.S. adult population from quota samples is generally inadvisable because they may still differ from the target population on unobserved characteristics. 8 However, a large body of prior work has demonstrated that survey experimental estimates obtained from nonprobability samples (including undergraduates and Amazon Mechanical Turk workers) closely match those from probability samples, demonstrating that results from nonprobability samples typically generalize to the broader population (Berinsky et al, 2012;Coppock, 2019;Coppock et al, 2018;Coppock & McClellan, 2019;Mullinix et al, 2015;Peyton et al, 2021). Our primary focus in this paper is therefore on the inferences that can be made from the survey experimental estimates we obtain.…”

“…Following best practices to ensure data quality in online research, we restricted participation to respondents who read and passed an attention screener placed at the beginning of the survey (see Aronow et al, 2020;Peyton et al, 2021 for recent work on respondent attentiveness, and Online Appendix Sections C.1 and D.1 for additional details). Of the 2099 respondents who consented to participate in the survey, 1137 passed the attention check and completed the survey.…”

“…5 For example, "defund the police" was mentioned more than 300 times in the 2 weeks following Floyd's death and the editorial boards of several major newspapers, including the Washington Post and Wall Street Journal, had published positions of support (or opposition) by early June 2020 (see Hollar, 2020). 6 In 2020, Lucid sold over 2 million completed survey responses to academic clients (see Peyton et al, 2021). 7 Our estimates are based on respondents' answers to the following survey experimental question: "Do you oppose or support [abolishing/defunding/reforming] the police?"…”

Ranked-Choice Voting and the Potential for Improved Electoral Performance of Third-Party Candidates in America

“…Questions about the data produced during the COVID-era, whether it is reliable and generalizable due to the extraordinary circumstances, emerged during the height of the pandemic. In response, Peyton et al (2020) replicated data from multiple online studies to analyze whether people's political preferences could have shifted and if the pandemic changed the types of respondents. Their results show that there could be an increase in "inattentive survey subjects (Peyton et al, 2020, 16)."…”

Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments

“…Lucid is an increasingly popular alternative to Amazon Mechanical Turk for social science survey research. Many well-known findings have been replicated on Lucid, suggesting the platform is capable of providing high-quality data (Coppock and McClellan 2019;Peyton, Huber, and Coppock 2021). During the COVID-19 pandemic, Ternovski and Orr (2022) found that Lucid data can provide reliable data when researchers screen on attentiveness, which we do here.…”