Crowd Access Path Optimization: Diversity Matters

Abstract

Quality assurance is one the most important challenges in crowdsourcing. Assigning tasks to several workers to increase quality through redundant answers can be expensive if asking homogeneous sources. This limitation has been overlooked by current crowdsourcing platforms resulting therefore in costly solutions. In order to achieve desirable cost-quality tradeoffs it is essential to apply efficient crowd access optimization techniques. Our work argues that optimization needs to be aware of diversity and correlation of information within groups of individuals so that crowdsourcing redundancy can be adequately planned beforehand. Based on this intuitive idea, we introduce the Access Path Model (APM), a novel crowd model that leverages the notion of access paths as an alternative way of retrieving information. APM aggregates answers ensuring high quality and meaningful confidence. Moreover, we devise a greedy optimization algorithm for this model that finds a provably good approximate plan to access the crowd. We evaluate our approach on three crowdsourced datasets that illustrate various aspects of the problem. Our results show that the Access Path Model combined with greedy optimization is cost-efficient and practical to overcome common difficulties in large-scale crowdsourcing like data sparsity and anonymity.

Human-in-the-loop machine learning with applications for population health

“…Therefore, weight-based aggregation methods are proposed, where the label of a worker with high weight would count more. Aggregation methods that consider the diversity and dependency of workers are proposed in recent years to improve the label aggregation accuracy [Nushi 2015] [Venanzi 2016]. In addition to the aggregation approach, worker selection and task assignment is also important.…”

Skill-and-Stress-Aware Assignment of Crowd-Worker Groups to Task Streams

“…Another research proposed producing event reports by using a combination of local and remote workers (Agapie, Teevan, and Monroy-Hernández 2015). Nushi et al (2015) considered the diversity of workers to avoid crowdsourcing redundancy. We also solve the convergence of the same types of workers in groups, but our work focuses on worker skills rather than worker types, which can optimize the worker-group assignment even in the case when workers come from homogeneous sources.…”

Learning and Feature Selection under Budget Constraints in Crowdsourcing

“…The goal is to learn a good classifier on a given budget. This section introduces the two models based on which we build our algorithm: Naïve Bayes model (Lowd and Domingos 2005), and Access Path model (Nushi et al 2015) as a representative of models that handle crowdsourcing errors. Later, we formally define our problem in the context of these two models.…”