Competitive Game Designs for Improving the Cost Effectiveness of Crowdsourcing

Rokicki, Markus; Chelaru, Sergiu; Zerr, Sergej; Siersdorfer, Stefan

doi:10.1145/2661829.2661946

Cited by 28 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Improving Effectiveness. Several optimization techniques have been introduced in prior works in order to increase the throughput of crowdworkers, maximize the cost-benefit ratio of deploying crowdsourced microtasks [45,46], and improving the overall effectiveness of the microtask crowdsourcing model. Gamification has been shown to improve worker retention and throughput of tasks [12].…”

Section: Existing Platforms Demand Workarounds -Current Solutionsmentioning

confidence: 99%

Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd

Gadiraju

Möller

Nöllenburg

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Section: Existing Platforms Demand Workarounds -Current Solutionsmentioning

confidence: 99%

Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd

Gadiraju

Möller

Nöllenburg

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

“…A second group of empirical work focused on bespoke experimental setups for crowdsourcing. For example, Reference [60] looked at the effect of varying monetary schemes and information policies in individual contests, while Reference [61] explored the same problem alongside team formation strategies. Both papers bear similarities to our scenario, in which contestants strategically decide whether they continue to take on more tasks or leave Wordsmith.…”

Section: Contest-basedmentioning

confidence: 99%

“…A worker would always seek to maximise their expected utility given the number and value of prizes; the number of contestants; and the value of their efforts. In our experiments, it was possible for workers to view their ranked position in real-time with respect to their closest contenders using a k-neighbours leaderboard, as presented in the medium information policy contest strategy by Reference [60]. A worker far outside the reward spread might inadvertently decide to exit the contest to avoid further loss of utility.…”

Section: Task Elementsmentioning

confidence: 99%

Beyond Monetary Incentives

Feyisetan

Simperl

2019

Trans. Soc. Comput.

View full text Add to dashboard Cite

More than a decade after the term was introduced by Jeff Howe in 2006 [33], crowdsourcing seems to be considered by many as a, if not "the," universal means to solve virtually any kind of problem, online and offline, that requires sustained human involvement. We see it used to motivate employees to engage with less rewarding work routines, attract the best possible ideas to boost innovation, enhance artificial intelligence algorithms, and support ambitious social and entrepreneurial initiatives. Achieving a goal by collecting contributions from many individuals has a long tradition, far beyond the most recent developments in the digital world. In fact, some of the most successful exemplars of the "wisdom of the crowds" in modern times, for instance Wikipedia, predate Howe's article. And yet, with the rise of social networks, smart mobile devices and online platforms, the phenomenon has found a new dimension in terms of scale and achievements-it routinely manages to mobilise very large groups of people in a relatively short period of time, helping organisations from tech and government to the military and marketing improve the ways they operate, decide, and engage with the world. The crowdsourcing landscape is as diverse as its applications. It includes paid microtask platforms such as Amazon's Mechanical Turk 1 and CrowdFlower (now Figure Eight), 2 alongside online labour marketplaces such as TaskRabbit 3 and UpWork, 4 and open innovation contests in the style of Kaggle 5 and Innocentive. 6 It features volunteer citizen science systems such as Galaxy Zoo 7 and SciStarter, 8 and games with a purpose (GWAPs) such as PhraseDetective 9 and EyeWire. 10 Putting aside the principled differences between these forms of crowdsourcing, which make them from the offset amenable only to specific types of problems, the success of any crowdsourcing endeavour will depend on the ability of the "requester"-the person or institution reaching out to the crowdto benefit from crowd outputs and attract a critical mass of contributors. The frame conditions can be challenging as well: To be effective, crowdsourcing needs to provide a real alternative to existing solutions of the problem the requester is trying to solve, in terms of quality, timeliness, and costs. Getting this mixture right is not trivial, as it requires insight into a wide array of subjects from artificial intelligence and data analysis to user experience design and behavioural sciences [21, 22]. Aligning the motivations of the crowd with the goals of the requester and finding the right mix of rewards to drive purposeful crowd engagement are particularly critical. Existing crowdsourcing platforms already provide some level of support-whether paid microtasks, challenges, or citizen science projects, each of these forms of crowdsourcing makes some basic assumptions about what drives people to contribute to their "open call" [33], and offers sometimes bespoke incentives to encourage a certain style of behaviour. For example, people register on Mechanical Turk primarily for f...

show abstract

“…To make crowdsourcing solutions scale to large amounts of data, it is key to design solutions that will retain crowd workers longer in the crowdsourcing platform [13] and prioritize work execution over the crowd.…”

Section: Crowdsourcing Efficiencymentioning

confidence: 99%

Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing

et al. 2015

View full text Add to dashboard Cite

The notion of crowdsourcing existed long before the term itself was coined. The idea that small contributions from a group of individuals can be accumulated to accomplish some work or attain an objective has been observed in different realms of daily life for a few centuries. Over the years, crowdsourcing has emerged as a useful paradigm, showcasing the adage that "the whole is greater than the sum of its parts."The Web's emergence as a sociotechnical system has dramatically changed the scale and scope of crowdsourcing, opening the possibility of reaching crowds at an unprecedented global scale. Crowdsourcing has proven to be an increasingly important source of knowledge and data, as documented by prominent examples such as Wiki-pedia, where many authors contribute discrete and diverse information to form an authoritative reference knowledge base. The recent Wiki-data project (www.wikidata.org) and the popular ReCAPTCHA work by Luis von Ahn and colleagues [1] Particularly in research and science, crowd-sourcing has found noteworthy applications in solving real-world problems, ranging from intricate tasks such as protein folding and bio-molecule design [2] and mapping outer space [3] to aiding disaster relief initiatives [4] and assembling dictionaries. [5] © 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. With the ubiquity of the Internet, and the concomitant accessibility of established microtask crowdsourcing platforms such as Amazon's Mechanical Turk (MTurk; www.mturk.com) and, more recently, CrowdFlower (www.crowdflower.com), researchers and practitioners are actively turning toward paid crowdsourcing to solve data-centric tasks that require human input. Popular applications include building ground truths, validating results, and curating data. These developments make it possible to build novel intelligent systems that combine the scalability of machines over large amounts of data with the power of human intelligence to solve complex tasks, such as audio transcription, language translation, and an-notation. For the rest of this article, crowdsourcing will refer to paid microtask crowdsourcing (wherein crowd workers receive monetary compensation for successfully completing a micro-task). For a thorough discussion of related terms and tasks in the context of human computation and collective intelligence, see the work of Alexander Quinn and Benjamin Bederson. [6] Owing to the diversity in the crowd in terms of workers' motivations, demographics, and competencies, both microtask design and quality control mechanisms play an unparalleled role in determining the effectiveness of crowdsourcing systems. [7] These two realms, which specifically are concerned with the requesters' perspecti...

show abstract

Competitive Game Designs for Improving the Cost Effectiveness of Crowdsourcing

Cited by 28 publications

References 21 publications

Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd

Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd

Beyond Monetary Incentives

Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing

Contact Info

Product

Resources

About