Qasca

Abstract

A crowdsourcing system, such as the Amazon Mechanical Turk (AMT), provides a platform for a large number of questions to be answered by Internet workers. Such systems have been shown to be useful to solve problems that are difficult for computers, including entity resolution, sentiment analysis, and image recognition. In this paper, we investigate the online task assignment problem: Given a pool of n questions, which of the k questions should be assigned to a worker? A poor assignment may not only waste time and money, but may also hurt the quality of a crowdsourcing application that depends on the workers' answers. We propose to consider quality measures (also known as evaluation metrics) that are relevant to an application during the task assignment process. Particularly, we explore how Accuracy and F-score, two widely-used evaluation metrics for crowdsourcing applications, can facilitate task assignment. Since these two metrics assume that the ground truth of a question is known, we study their variants that make use of the probability distributions derived from workers' answers. We further investigate online assignment strategies, which enables optimal task assignments. Since these algorithms are expensive, we propose solutions that attain high quality in linear time. We develop a system called the Quality-Aware Task Assignment System for Crowdsourcing Applications (QASCA) on top of AMT. We evaluate our approaches on five real crowdsourcing applications. We find that QASCA is efficient, and attains better result quality (of more than 8% improvement) compared with existing methods.

Understanding the Skill Provision in Gig Economy from A Network Perspective

“…In this context, early studies often consider only a single skill and solve problems like how a worker's capability on this skill can be accurately estimated [20], and how can the employers identify the most qualified worker for a skill through social networks [5]. More recently, an increasing number of studies have been conducted to examine scenarios where each worker possesses multiple skills [2,9,15,27,31,32]. Different algorithms have been proposed to match a task to the "right" worker who is most capable on the skill that the task requires, while the estimation of worker's capacity on each skill keeps updated in an online fashion.…”

Yang Chengzong and the University of Science and Technology of China

“…However, all the above methods suffer time complexity issues since they need to solve optimization problems during inference. Meanwhile, in real-world crowdsourcing platforms, multiple time efficient functions like task assignment and convergence monitoring need frequent access to the temporal inference results Zheng et al [2015], so that previous methods can not fully satisfy the real-world requirements. INQUIRE Feng et al [2014] first addresses this problem via a trivial annotator modeling strategy and a straightforward weighted vote question model to incrementally infer truths and update models.…”

A Model for Cognitive Personalization of Microtask Design

“…For instance, a study on quality control mechanisms proposed a set of indicators and a general framework covering more types of microtask, including open-ended answers [ 27 ]. This work obtained better outcomes when compared to the state-of-the-art methods, such as the traditional analysis of historical performance in crowd work [ 28 ]. Furthermore, a supervised ML model was proposed, where more types of crowd worker profile were detected, with a higher granularity [ 29 ].…”