Amazon Mechanical Turk: Gold Mine or Coal Mine?

Abstract

Last Words editorial for Computational LinguisticsInternational audienceN/

A streamlined approach to online linguistic surveys

“…In all the AMT experiments conducted at the Experimental Syntax-Semantic Lab at MIT between November 2010 and April 2013, a total of 4635 unique workers participated in at least one study; 643 (14%) participated in more than one study, 152 (3%) participated in more than two studies, and only 15 (0.3%) participated in more than five. 6 Thus, we believe that the diversity of our data is not jeopardized by the tendencies described in Fort et al (2011). 7 5 A screen capture of this map can be found at http://turktools.net/crowdsourcing/.…”

“…We refer the reader to the worker map in Tamir (2011), which details the country of residence of 50,000 AMT workers. 5 We note that some concerns have been raised about the nature of the AMT participant pool in the context of linguistic experiments, in particular as to the proportion of workers who complete most of the tasks posted on AMT (Fort et al 2011). From our own experience with AMT, however, we believe that such concerns are unfounded.…”

“…Requesters generally pay very little per task and retain the ability to Accept or Reject the results of each task before AMT sends payment to the worker. Tasks can be posted using an online interface (http://mturk.com), and their results are easily analyzable using mainstream statistical software (for more information, see Fort et al 2011;Gibson et al 2011;Sprouse 2011). …”

Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments

“…Fort et al [40] further analyzed the above data and presented more details on the task distribution. According to the authors, about 80% of the tasks in AMT were carried out by less than 10,000 Turkers, which represented roughly one percent of the registered crowd-workers at that time.…”

On the Ethics of Crowdsourced Research

“…In the fi eld of computational linguistics, Fort, Adda, and Cohen ( 2011 ) suggest that researchers fi nd alternatives to crowdsourcing. Where MTurk is used to mimic machine labor, this may provide an obvious solution to the problem of crowdsource exploitation.…”