Mining Crowdsourcing Problems from Discussion Forums of Workers

Abstract

Crowdsourcing is used in academia and industry to solve tasks that are easy for humans but hard for computers, in natural language processing mostly to annotate data. The quality of annotations is affected by problems in the task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. To learn about the major problems, we provide a short but comprehensive survey based on two complementary studies: (1) a literature review where we collect and organize problems known from interviews with workers, and (2) an empirical data analysis where we use topic modeling to mine workers' complaints from a new English corpus of workers' forum discussions. While literature covers all process phases, problems in the task evaluation are prevalent, including unfair rejections, late payments, and unjustified blockings of workers. According to the data, however, poor task design in terms of malfunctioning environments, bad workload estimation, and privacy violations seems to bother the workers most. Our findings form the basis for future research on how to improve crowdsourcing processes. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. create task Platform (1) Task design (2) Task operation (3) Task evaluation assess results post task take on task solve task obtain results submit results accept or reject get paid or not questions, inquiries responses, feedback explanations, responses complaints Requester Workers Figure 1: The three phases of a crowdsourcing process. (1) Task design: The requester creates a task.(2) Task operation: Workers accept and solve the task, and submit the results.(3) Task evaluation: The requester accept or reject the results. Communication may happen during task operation and evaluation.approaches in this paper, bringing together knowledge from human computation with ideas from NLP: First, a literature review where we collect crowdsourcing problems discussed in related work, mostly obtained from interviews with and surveys among workers (Section 3). We consolidated the reported challenges from a wide range of studies in order to form a unified view on problems already known in literature. Second, an empirical data analysis where we use topic modeling to mine the workers' problems from what they complain about on a daily basis in online discussion forums (Section 4). Given its dominance, we focus on MTurk here, building a new corpus of 27,041 reviews from Turkopticon, a reputation portal where MTurk workers write reviews on requesters. In our analysis, we focus on the 8,610 negative reviews, since they are likely to contain complaints. 1 The hypothesis underlying our combined approach is that the literature gives a more comprehensive view of problems existing in crowdsourcing, whereas the data better reflects their significance. In light Rof this hypothesis, we study two research questions:RQ1. What problems do workers face in the different phases of crowdsour...

What Is Unclear? Computational Assessment of Task Clarity in Crowdsourcing

“…Consequently, achieving task clarity is of great importance to crowdsourcing processes. Yet, prior research on crowdsourcing emphasizes that ambiguous task descriptions have been a constant challenge [13,23,27,28,38,50,58]. On one hand, requesters are required to adequately describe all information necessary for completing a given task, including the resources to use, the steps to be followed, and the solution format to submit.…”

“…This flourishing paradigm provides the potential to harness the wisdom, abilities, and creativity of a crowd for problems that require human intelligence. The general crowdsourcing process includes three main phases [50]: (1) Task design, where requesters post task descriptions on a crowdsourcing platform. (2) Task operation, where workers accept tasks and then submit their results.…”

“…Lowquality solutions are considered as the major barrier to achieve the full potential of crowdsourcing [57]. This problem emerges from challenges pertaining to the three main stakeholders involved: (1) workers may be unqualified, inexperienced, or unmotivated to complete tasks effectively [16,31,36,52,58]; (2) requesters may also be inexperienced, unfair, or sometimes careless in task design, task operation, and task evaluation [11,36,50,58]; and (3) the platform may mediate the entire crowdsourcing process poorly, especially in terms of facilitating requester-worker communication [50]. Among several factors that have been shown to shape the quality of crowd work, unclear task design has been highlighted as one of the most critical [27,41,43,48,58].…”

A Video Game-Crowdsourcing Approach to Discover a Player’s Strategy for Problem Solution to Housing Development

“…From this idea surged a concept called Crowdsourcing. Instead of collecting data that generates daily, Crowdsourcing achieves solutions using the skills of a huge crowd of volunteers [2], [3]. Its objective is to broadcast a specific problem to individuals with certain knowledge [4].…”

“…Their objective is to detect novel, high-quality solutions; therefore, the range of problems it can cover is wide. Some recent applications include smart cities [5], genetic variability [6] and Natural Language Processing research that take advantage of applications like Amazon Mechanical Turk [3]. Very rarely, though, crowdsourcing is used to uncover an understanding of the decision-making process and strategizing that users follow during problem-solving.…”