Mechanical Turk data collection in addiction research: utility, concerns and best practices

Abstract

Aims Amazon Mechanical Turk (MTurk) provides a crowdsourcing platform for the engagement of potential research participants with data collection instruments. This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research. Methods We review four use cases of MTurk for research relevant to addictions: (1) the development of novel measures, (2) testing interventions, (3) the collection of longitudinal use data to determine the feasibility of longer‐term studies of substance use and (4) the completion of large batteries of assessments to characterize the relationships between measured constructs. We review concerns with the platform, ways of mitigating these and important information to include when presenting findings. Results MTurk has proved to be a useful source of data for behavioral science more broadly, with specific applications to addiction science. However, it is still not appropriate for all use cases, such as population‐level inference. To live up to the potential of highly transparent, reproducible science from MTurk, researchers should clearly report inclusion/exclusion criteria, data quality checks and reasons for excluding collected data, how and when data were collected and both targeted and actual participant compensation. Conclusions Although on‐line survey research is not a substitute for random sampling or clinical recruitment, the Mechanical Turk community of both participants and researchers has developed multiple tools to promote data quality, fairness and rigor. Overall, Mechanical Turk has provided a useful source of convenience samples despite its limitations and has demonstrated utility in the engagement of relevant groups for addiction science.

Are you for real? Maximizing participant eligibility on Amazon's Mechanical Turk

“…Also, as the interest in developing efficacious on-line interventions for addictions grows [2,3], Mechanical Turk (MTurk) presents a viable platform for the development and testing of these interventions due to its affordability and large participant pool. Nevertheless, its use in clinical research comes with challenges and limitations, such as non-naiveté, worker inattention and fraudulent responses [1,4]. This commentary explores other challenges with conducting research on MTurk, including those specific to intervention testing, and expands upon what has been addressed by Mellis & Bickel [1].…”

“…Nevertheless, its use in clinical research comes with challenges and limitations, such as non-naiveté, worker inattention and fraudulent responses [1,4]. This commentary explores other challenges with conducting research on MTurk, including those specific to intervention testing, and expands upon what has been addressed by Mellis & Bickel [1].…”

“…None. As more researchers turn to on-line convenience samples in an effort to reach hidden populations of substance users, reviews such as that by Mellis & Bickel [1] are both timely and necessary. Also, as the interest in developing efficacious on-line interventions for addictions grows [2,3], Mechanical Turk (MTurk) presents a viable platform for the development and testing of these interventions due to its affordability and large participant pool.…”

“…Indeed, this issue has been visited within the literature numerous times, and several strategies have been offered to minimize its occurrence (e.g. checking for common virtual private servers, response consistency, random responding and unique IP addresses) [1,5,6]. However, another facet of this problem is the sharing of valuable study information among MTurk workers on forums.…”

“…Mellis & Bickel [1] provide an overview of Mechanical Turk (MTurk) data collection along with best practice guidelines. Our laboratory has had success in using MTurk for the past 6 years, including some methodological examinations for addictions research (e.g.…”