Are poor quality data just random responses?: A crowdsourced study of delay discounting in alcohol use disorder.

Abstract

Crowdsourced methods of data collection such as Amazon Mechanical Turk (MTurk) have been widely adopted in addiction science. Recent reports suggest an increase in poor quality data on MTurk, posing a challenge to the validity of findings. However, empirical investigations of data quality in addiction-related samples are lacking. In this study of individuals with alcohol use disorder (AUD), we compared poor quality delay discounting data to randomly generated data. A reanalysis of prior published delay discounting data was conducted comparing included, excluded, and randomly generated data samples. Nonsystematic criteria were implemented as a measure of data quality. The excluded data was statistically different from the included sample but did not differ from randomly generated data on multiple metrics. Moreover, a response bias was identified in the excluded data. This study provides empirical evidence that poor quality delay discounting data in an AUD sample is not statistically different from randomly generated data, suggesting data quality concerns on MTurk persist in addiction samples. These findings support the use of rigorous methods of a priori defined criteria to remove poor quality data post hoc. Additionally, it highlights that the use of nonsystematic delay discounting criteria to remove poor quality data is rigorous and not simply a way of removing data that does not conform to an expected theoretical model. Public Health SignificanceThis study provides empirical evidence that poor quality delay discounting data does not differ from random responding in a sample of individuals with alcohol use disorder. This highlights that previous reports of poor quality data on Amazon Mechanical Turk extend to addiction-related samples. Thus, the use of rigorous data quality controls and exclusion of poor quality data are warranted to ensure highquality scientific findings.

The relationship between nonsystematic delay discounting and low-quality survey responses in a sample of smokers: ROC curve analysis.

“…A key finding of this study is that nonsystematic responding during a DD task was a predictor of low-quality data across three independent quality check indexes (i.e., reporting daily cigarette usage with consistency; describing a choice task accurately; reporting individual and household income with consistency). A previous investigation in AUD reported that nonsystematic DD data were similar to randomly generated data on multiple metrics (Craft et al, 2022). The current finding supports the connection between nonsystematic DD and low-quality data.…”

Delay discounting in adults with and without chronic pain: Differentiation across commodity and sign.

“…Violation of the second criterion suggests that reward valuation is not affected by delay. Excluding nonsystematic data is widely used in the field (Smith et al, 2018), and recent investigations have demonstrated that nonsystematic delay discounting is associated with poor data quality and random responding in online samples (Craft et al, 2022; Yeh et al, 2022). Prior to conducting analyses, we determined that 19 individuals failed attention check questions and 14 individuals violated one or more nonsystematic delay discounting criteria.…”

Crowdsourcing methods in addiction science: Emerging research and best practices.

“…Conceptually related behavioral economic research shows the practical implications of these quality checks. Craft et al (2022) find that delay-discounting data from participants failing systematicity checks did not differ from randomly generated data highlighting the importance of screening and removing data with a priori validity checks. Freitas-Lemos et al (2022) describe how a novel check based on instructional understanding differentiated participants on consistency of cigarette use reporting, responding on a cigarette demand task, and the relationship between use behavior and demand data.…”