Managing crowdsourced human computation

Abstract

The proposed tutorial covers an emerging topic of wide interest: Crowdsourcing. Specifically, we cover areas of crowdsourcing related to managing structured and unstructured data in a web-related content. Many researchers and practitioners today see the great opportunity that becomes available through easily-available crowdsourcing platforms. However, most newcomers face the same questions: How can we manage the (noisy) crowds to generate high quality output? How to estimate the quality of the contributors? How can we best structure the tasks? How can we get results in small amounts of time and minimizing the necessary resources? How to setup the incentives? How should such crowdsourcing markets be setup?Their presented material will cover topics from a variety of fields, including computer science, statistics, economics, and psychology. Furthermore, the material will include real-life examples and case studies from years of experience in running and managing crowdsourcing applications in business settings.The tutorial presenters have an extensive academic and systems building experience and will provide the audience with data sets that can be used for hands-on tasks.Keywords crowdsourcing mechanical turk workflow control quality assurance incentives reputation market design human computation

Two's company, three's a crowd: a case study of crowdsourcing software development

“…A key issue in crowdsourcing is that work is decomposed into a set of smaller tasks [45,52,54]. This issue is highly relevant in outsourcing scenarios, and Herbsleb and Grinter [36] reminded us of Parnas' definition of a module as "a responsibility assignment rather than a subprogram" [64].…”

“…At the same time, there is a risk of 'noise' in submissions, where solutions are of a low quality [24,45]. In a software development context, the idea that input from a wide variety of developers helps in finding and fixing defects is better known as Linus's Law, or, "given enough eyeballs, all bugs are shallow" [66].…”

“…A final consideration in crowdsourcing is that of motivation and remuneration [19,22,29,40,45,59,60,65]. Motivation is a topic that has received considerable attention in the software engineering research field, given that it is reported to be a major factor in project success [6,11].…”

Which, When, and How: Hierarchical Clustering with Human–Machine Cooperation

“…Recently, combining human and machine intelligence in crowdsourcing [12] has become a hot research area. When the machine cannot solve its task, it can ask for human help.…”

Researching crowdsourcing software development: perspectives and concerns

“…In order to benefit from a potentially large crowd, the system should be split up into many small pieces that can be developed in parallel by different developers in the crowd. This raises an old question in software engineering, namely, how should the system be decomposed into smaller modules without causing problems in putting them back together once they are developed [18,20,21]. Common questions in software engineering within the scope of decomposition relate to assumptions, interfaces and dependencies.…”