Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.

Substantial differences in attentional disengagement between Prolific and MTurk samples: Implications for experimental outcomes

“…Fewer reports of mind wandering, media-related multitasking, and higher task performance provide convergent evidence for greater attentional engagement in the Proli c sample compared to the MTurk sample. These ndings corroborate observations from other studies showing signi cantly greater engagement in Proli c samples as indicated by more correct responses to attention check questions 38,41,54 . A novel contribution of the present study, however, is its use of attention task performance and self-reports of disengagement, in terms of mind wandering and multitasking during an ongoing task to assess attentional engagement, instead of relying on attention to survey instructions or question wording, as in previous studies.…”

Catching a smile from individuals and crowds: Evidence for distinct emotional contagion processes.

“…Further replications are, nevertheless, advised. We also acknowledge that Study 5 is the only one that was carried out online, and while attention checks were included to assess fidelity and research indicates that data captured via Prolific are of good quality (Douglas et al, 2023), this does remove some of the methodological controls present during in-person testing. Future replications are therefore recommended, particularly to examine whether Study 5 findings translate into objective measures of affect.…”

Numeric social-media posts engage people with climate science

“…About 50 participants were required for the main effect of tweet type; we then oversampled to ensure enough participants to test interaction effects and covariates. CloudResearch was chosen due to high data quality in our own past studies and higher quality relative to other data vendors 43 . To ensure high quality participants, we recruited participants to the baseline cohort by requiring a 99% -100% HIT approval rating and completion of at least 1,000 HITS, age over 18, and being within the United States.…”