The Dynamics of Micro-Task Crowdsourcing

Difallah, Djellel Eddine; Catasta, Michele; Demartini, Gianluca; Ipeirotis, Panagiotis G.; Cudré-Mauroux, Philippe

doi:10.1145/2736277.2741685

Cited by 159 publications

(73 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The 2010 report by Ipeirotis noted that the most prevalent types on AMT are transcription, data collection, image tagging, and classification [37]. Follow-up work by Difallah et al reaffirms these findings, although tasks like audio transcription are becoming more prevalent [21].…”

Section: Terminologymentioning

confidence: 99%

“…Prior research based on self-reported income data (e.g., [4,34,49]) might be subject to systemic biases [22] and is often not sufficiently granular to facilitate a detailed investigation of earnings dispersion. Existing data-driven quantitative work in crowdsourcing research has taken the employers' perspective [49] (e.g., finding good pricing methods [36,50,61], suggesting effective task design for requesters [24,40]), or it characterizes crowdsourcing market dynamics [21,37]. Data-driven research on how workers are treated on the markets is missing.…”

Section: Introductionmentioning

confidence: 99%

“…Our dataset includes the records of 3.8 million HITs that were submitted or returned by 2,676 unique workers. The data includes task duration and HIT reward, which allows us to evaluate hourly wage rates-the key measure that has been missing from the prior data-driven research [21,40]-at an unprecedented scale. We provide the first task-level descriptive statistics on worker earnings.…”

Section: Introductionmentioning

confidence: 99%

“…Existing quantitative work in crowdsourcing research has taken the employers' perspective [49] (e.g., finding good pricing methods [36,50,61], suggesting effective task design for requesters [24,40]) or it focuses on characterizing the crowdsourcing market dynamics [21,37]. Although important, data-driven research on how workers are treated on the crowdsourcing markets is missing.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

366

266

View full text Add to dashboard Cite

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work. Figure 12. (a) Hourly wage distributions of seven HIT categories provided by Gadiraju et al. [25] (with an additional category Research). (b) Strip plots showing median hourly wages of HITs associated with the topical keywords in Table 4.

show abstract

Section: Terminologymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

366

266

View full text Add to dashboard Cite

show abstract

“…Crowdsourcing platforms are rapidly gaining ground in the scientific community (Difallah et al, ), as they allow for the recruitment and testing of many highly diverse participants across wide geographic ranges in short time frames. Amazon’s Mechanical Turk (MTurk) is a common crowdsourcing platform used by a growing number of psychologists and other researchers.…”

Section: Methodsmentioning

confidence: 99%

Wake Forest Alcohol Imagery Set: Development and Validation of a Large Standardized Alcohol Imagery Dataset

Peterson

Simpson

Laurienti

2019

Alcoholism Clin & Exp Res

View full text Add to dashboard Cite

Background: The study of alcohol use frequency utilizes alcohol-related cue imagery. Although a number of alcohol-image databases currently exist, they have several limitations: Many are not publicly available, some use stock images or clip art rather than real photographs, several eliminate any photographs displaying brand information, and predominantly they contain relatively few images. The aim of this project was to develop a large, open-access database of alcohol-related cue images, containing photographs with and without brand information, taken in real-world environments, with images in a variety of orientations and dimensions.Methods: The study collected 1,650 images voluntarily from the larger community, to capture photographs with a wide range of content, environments, and relation to alcohol. All images were then rated on scales of valence, arousal, and relation to alcohol by 1,008 Amazon Mechanical Turk workers, using classical emotion validation methods based on the International Affective Picture System (IAPS). Survey respondents were screened with the Alcohol Use Disorders Identification Test (AUDIT), and Cronbach's alpha scores were calculated to determine the interrater reliability of scores across the whole sample, and within low-risk, moderate-risk, and high-risk drinkers for each rating domain. Univariate ANOVAs were run to determine differences in ratings across drinking groups.Results: All Cronbach's alpha scores indicated high interrater reliability within the whole sample, and across drinking severity groups. Tukey's HSD post hoc results indicated greater arousal and affect in response to image viewing in moderate-and high-risk drinkers, and higher relation-to-alcohol ratings in low-risk drinkers. All images had categorization tags assigned by members of the study team.Conclusions: The established imagery set includes 1,650 alcohol-related images, rated on scales of valence, arousal, and relation to alcohol, and categorized by type of alcohol depicted. The imagery database will be available for open-access download and use through Google Photos.

show abstract