Tapped Out or Barely Tapped? Recommendations for How to Harness the Vast and Largely Unused Potential of the Mechanical Turk Participant Pool

Abstract

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool.Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

Counting ‘micro-workers’: societal and methodological challenges around new forms of labour

“…This is less than the advertised population, but almost fourteen times as many as Stewart et al (2015) claimed. A more recent study gets even closer to Amazon's own figures, finding 250,000 Turkers worldwide using data from TurkPrimer, an independent company that helps researchers design and implement studies on Mechanical Turk (Robinson, Rosenzweig, Moss & Litman, 2019).…”

Mechanical Turk data collection in addiction research: utility, concerns and best practices

“…Non-naivete • Post HITs in large batches, to increase the effective worker pool available •Assign qualifications to workers who have completed previous, similar HITs. Share these worker IDs among collaborating research groups [46], with participant consent •Allow workers with a lower number of completed HITs to complete assessments [47]. Note, however, that 'reputations' are not granted until a worker has completed > 100 HITs Worker inattention…”

“…Report full sample collected and reasons for data exclusion • Screen based on worker 'reputation' [40]. Note, however, that this common practice may reduce worker naivete [47] Fraudulent responses…”

Measuring delay discounting in a crowdsourced sample: An exploratory study

“…The present study had several limitations that provide directions for future research. First, our sample was a convenience sample crowdsourced on Amazon Mechanical Turk, which is a crowdsourcing website commonly used by researchers across disciplines to collect large amounts of quality data for relatively small costs (Amir et al ., ; Bohannon, ; Buhrmester, Kwang & Gosling, ; Chandler & Shapiro, ; Goodman & Paolacci, ; Robinson, Rosenzweig, Moss & Litman, ). Although our goal was to determine how to measure delay discounting in this population, and results using such populations may be generalizable to other online users who do tasks for relatively small amounts of money, we cannot generalize to other populations who are accustomed to doing tasks or making decisions involving larger amounts of money.…”