The Dynamics of Micro-Task Crowdsourcing

Difallah, Djellel Eddine; Catasta, Michele; Demartini, Gianluca; Ipeirotis, Panagiotis G.; Cudré-Mauroux, Philippe

doi:10.1145/2740908.2744109

Cited by 72 publications

(72 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It provides excellent statistics, but covers conventional employment only, not gig online work. Similarly, the MTurk Tracker project (Difallah et al, 2015;Ipeirotis, 2010) tracks new and completed tasks on Amazon Mechanical Turk, an online labour platform. It produces interesting statistics in almost real time, but is limited to a single platform that is not a very good representative of the online gig economy more generally.…”

Section: Introductionmentioning

confidence: 99%

Online labour index: Measuring the online gig economy for policy and research

Kässi

Lehdonvirta

2018

Technological Forecasting and Social Change

331

232

View full text Add to dashboard Cite

Abstract. Labour markets are thought to be in the midst of a dramatic transformation, where standard employment is increasingly supplemented or substituted by temporary gig work mediated by online platforms. Yet the scale and scope of these changes is hard to assess, because conventional labour market statistics and economic indicators are ill-suited to measuring online gig work. We present the Online Labour Index (OLI), a new economic indicator that provides the online gig economy equivalent of conventional labour market statistics. It measures the utilization of online labour across countries and occupations by tracking the number of projects and tasks posted on platforms in near-real time. We describe how the OLI is constructed and demonstrate how it can be used to address previously unanswered questions about the online gig economy; in particular, we show that the online gig economy grew at an annualized rate of 14 percent. To benefit policymakers, labour market researchers, and the general public, the index is available as an open data set and interactive online visualization, which are automatically updated daily.

show abstract

Section: Introductionmentioning

confidence: 99%

Online labour index: Measuring the online gig economy for policy and research

Kässi

Lehdonvirta

2018

Technological Forecasting and Social Change

331

232

View full text Add to dashboard Cite

show abstract

“…Workers have identified improper tasks in a Japanese crowdsourcing site [17] and proofread documents in near-real time [4]. Other researchers were interested in analyzing the demographics of workers [18] and quantifying the evolution of campaigns/tasks in MTurk [19]. Ge et al analyzed a supply-driven crowdsourcing marketplace regarding key features that distinguish "super sellers" from regular participants [20].…”

Section: Related Workmentioning

confidence: 99%

Detecting malicious campaigns in crowdsourcing platforms

Choi¹,

Lee²,

Webb³

2016

2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)

View full text Add to dashboard Cite

Crowdsourcing systems enable new opportunities for requesters with limited funds to accomplish various tasks using human computation. However, the power of human computation is abused by malicious requesters who create malicious campaigns to manipulate information in web systems such as social networking sites, online review sites, and search engines. To mitigate the impact and reach of these malicious campaigns to targeted sites, we propose and evaluate a machine learning based classification approach for detecting malicious campaigns in crowdsourcing platforms as a first line of defense, and build a malicious campaign blacklist service for targeted site providers, researchers and users.Specifically, we (i) conduct a comprehensive analysis to understand the characteristics of malicious campaigns and legitimate campaigns in crowdsourcing platforms, (ii) propose various features to distinguish between malicious campaigns and legitimate campaigns, (iii) evaluate a classification approach against baselines, and (iv) build a malicious campaign blacklist service. Our experimental results show that our proposed approaches effectively detect malicious campaigns with low false negative and false positive rates. Web. To solve this problem, we build a predictor that detects whether campaign/task in crowdsourcing sites is malicious or not so that malicious campaigns/tasks can be removed by crowdsourcing site providers as soon as they are created. In particular, we (i) analyze characteristics of malicious and legitimate campaigns; (ii) extract commonly available features from four crowdsourcing sites; and (iii) build predictors and evaluate their performance. Our experimental results show that our predictors are more effective and robust compared with several baselines. In the end, we design and build a malicious campaign blacklist service, which provides users with various information.

show abstract

“…There has been a constant evolution of its usage patterns since it was launched in 2005. We presented a large-scale analysis of log data from this micro-task crowdsourcing platform, [9] showing how the use of microtask crowd-sourcing has evolved over the past five years.…”

Section: Microtask Evolution Over the Yearsmentioning

confidence: 99%

“…[8]- [9][10] Although crowdsourcing presents many open challenges, including ethical concerns we focus on performance-related aspects. In this article, we provide an overview of frequently crowdsourced microtasks, malicious activity observed in crowds, and open research challenges in the field.…”

mentioning

confidence: 99%

Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing

et al. 2015

Self Cite

View full text Add to dashboard Cite

The notion of crowdsourcing existed long before the term itself was coined. The idea that small contributions from a group of individuals can be accumulated to accomplish some work or attain an objective has been observed in different realms of daily life for a few centuries. Over the years, crowdsourcing has emerged as a useful paradigm, showcasing the adage that "the whole is greater than the sum of its parts."The Web's emergence as a sociotechnical system has dramatically changed the scale and scope of crowdsourcing, opening the possibility of reaching crowds at an unprecedented global scale. Crowdsourcing has proven to be an increasingly important source of knowledge and data, as documented by prominent examples such as Wiki-pedia, where many authors contribute discrete and diverse information to form an authoritative reference knowledge base. The recent Wiki-data project (www.wikidata.org) and the popular ReCAPTCHA work by Luis von Ahn and colleagues [1] Particularly in research and science, crowd-sourcing has found noteworthy applications in solving real-world problems, ranging from intricate tasks such as protein folding and bio-molecule design [2] and mapping outer space [3] to aiding disaster relief initiatives [4] and assembling dictionaries. [5] © 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. With the ubiquity of the Internet, and the concomitant accessibility of established microtask crowdsourcing platforms such as Amazon's Mechanical Turk (MTurk; www.mturk.com) and, more recently, CrowdFlower (www.crowdflower.com), researchers and practitioners are actively turning toward paid crowdsourcing to solve data-centric tasks that require human input. Popular applications include building ground truths, validating results, and curating data. These developments make it possible to build novel intelligent systems that combine the scalability of machines over large amounts of data with the power of human intelligence to solve complex tasks, such as audio transcription, language translation, and an-notation. For the rest of this article, crowdsourcing will refer to paid microtask crowdsourcing (wherein crowd workers receive monetary compensation for successfully completing a micro-task). For a thorough discussion of related terms and tasks in the context of human computation and collective intelligence, see the work of Alexander Quinn and Benjamin Bederson. [6] Owing to the diversity in the crowd in terms of workers' motivations, demographics, and competencies, both microtask design and quality control mechanisms play an unparalleled role in determining the effectiveness of crowdsourcing systems. [7] These two realms, which specifically are concerned with the requesters' perspecti...

show abstract

The Dynamics of Micro-Task Crowdsourcing

Cited by 72 publications

References 0 publications

Online labour index: Measuring the online gig economy for policy and research

Online labour index: Measuring the online gig economy for policy and research

Detecting malicious campaigns in crowdsourcing platforms

Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing

Contact Info

Product

Resources

About