To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing

Abstract

Digital games are a viable alternative to accomplish crowdsourcing tasks that would traditionally require paid online labor. This study compares the quality of crowdsourcing with games and paid crowdsourcing for simple and complex annotation tasks in a controlled exper-iment. While no difference in quality was found for the simple task, paid contributors’ response quality was sub-stantially lower than players’ quality for the complex task (92% vs. 78% average accuracy). Results suggest that crowdsourcing with games provides similar and potentially even higher response quality relative to paid crowdsourcing.

Performance of Paid and Volunteer Image Labeling in Citizen Science — A Retrospective Analysis

“…This payment scheme led to an average of USD 0.01226 per label, which is commensurate with other payment schemes, e.g. Krause and Kizilcec (2015) The workflow for workers was the same as volunteers, with the exception that workers did not have the option of revisiting the project. For the 2016 imagery (primary dataset), we recruited 156 workers in February 2022.…”

“…With simple tasks, Mao et al (2013) found that paying workers for time spent led to similar quality responses as volunteers (though accuracy could be traded for speed by switching to a "pay per task" model). Krause and Kizilcec (2015) found similar results, though they gave volunteers an image labeling game while giving workers a direct image labeling task.…”

“…Differences between response quality started to appear with higher difficulty tasks, however. When Krause and Kizilcec (2015) attempted to use workers and volunteers for a complex website annotation task, they noticed that volunteer responses were of higher quality than worker responses. Similarly, Sarkar and Cooper (2018) found higher quality responses with volunteers who played a human computation focused video game, as opposed to workers who were tasked with doing the same.…”

“…Our task is notably distinct from previous work in the arguably more specialized image labeling task. Whereas image labeling in some other works (Mao et al 2013;Krause and Kizilcec 2015) referred to associating or typing nouns that were represented in a photograph, our image labeling task required participants to identify subtle visual patterns that are related to wetland loss or restoration. Furthermore, we use a "recruitment cost" fixed payment scheme as opposed to paying per task / paying per time like Mao et al (2013) and Krause and Kizilcec (2015).…”

A longitudinal study of hacker behaviour

“…Other suggestions to improve productivity and bolster motivation include: having experts liaise with hackers, 'game-ifying' the searching and report process, and decomposing search domains into small task areas [11,34]. So-called 'gamification' of platforms currently exists in the form of leaderboards, badges, and reward points; however, it has been recognised that the extent to which such techniques are employed is limited [42].…”