Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool

Abstract

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk’s utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online

“…There are some aspects researchers should consider when starting out with running online studies or transferring lab-based experiments to online systems [59,75,76] (see Figure 2). To a certain extent, creating successful online experiments is similar to app development: one needs to think of a coherent framework and constantly worry about what the users are doing with the 'product' and whether they are using it as intended-without many opportunities for direct feedback.…”

Turking in the time of COVID

“…The fraction of participants giving at least one unexpected CRT response is 15% among baseline workers, 29% among pre-quarantine workers, 38% among post-quarantine workers, 19% among Prolific workers, and 34% among Lucid workers. Furthermore, the fraction of MTurk workers with at least one unexpected response in a 2019 study with no restrictions other than being located in the US was of 25% (5). Thus, new MTurk workers do not seem appreciably worse than participants from Lucid but they do seem to perform worse on the CRT than Prolific users and their MTurk counterparts a year ago.…”

“…Here we consider the impact of the current pandemic on subject pool composition by looking at 15,608 responses given to studies conducted by our group on MTurk between February and July 2020. Moreover, by exploiting a recent change in our recruitment approach, we are better suited to attract new participants, to track unrestricted and traditionally restricted samples over time (5) and, relative to usual recruitment restrictions, to anticipate eventual surgesas many of the new accounts considered would meet their criteria within a few weeks (6).…”

Increasing Profit Margins by Substituting Species in Floral Arrangements

“…Second, the sample size of this study is limited based on Krejcie and Morgan (1970), who recommended a sample size of 382 for a population size of more than 75,000. A most recent study (Robinson et al, 2019) indicated the Amazon Mechanical Turk panel, where our survey was distributed, have at least 250,810 workers worldwide and 226,500 workers in the United States. There were 80,000-85,000 active workers per year from 2016 to 2018 (Robinson et al, 2019).…”

“…A most recent study (Robinson et al, 2019) indicated the Amazon Mechanical Turk panel, where our survey was distributed, have at least 250,810 workers worldwide and 226,500 workers in the United States. There were 80,000-85,000 active workers per year from 2016 to 2018 (Robinson et al, 2019). Accordingly, the researchers can surmise that the generalizability of the results is constrained by the sample frame limitation.…”