Sequential multi-class labeling in crowdsourcing

Abstract

Abstract-We consider a crowdsourcing platform where workers' responses to questions posed by a crowdsourcer are used to determine the hidden state of a multi-class labeling problem. As workers may be unreliable, we propose to perform sequential questioning in which the questions posed to the workers are designed based on previous questions and answers. We propose a Partially-Observable Markov Decision Process (POMDP) framework to determine the best questioning strategy, subject to the crowdsourcer's budget constraint. As this POMDP formulation is in general intractable, we develop a suboptimal approach based on a q-ary Ulam-Rényi game. We also propose a sampling heuristic, which can be used in tandem with standard POMDP solvers, using our Ulam-Rényi strategy. We demonstrate through simulations that our approaches outperform a non-sequential strategy based on error correction coding and which does not utilize workers' previous responses.

Using Social Network Information in Community-Based Bayesian Truth Discovery

“…zn→m is the index of the community agent n subscribes to under the social influence of agent m. φ = ((φ n→m,k ) k )n,m β = (β k ) k β k is the social network parameter defined in (8).…”

Task Recommendation in Crowdsourcing Based on Learning Preferences and Reliabilities

“…Our work belongs to this category. In [7], [31], as workers may be unreliable, the authors proposed to perform sequential questioning in which the questions posed to the workers are designed based on previous questions and answers. Reference [32] studied sequential user selection using a Bayes update aided online solution in the problem of news disseminating.…”

“…They have shown their algorithm performs well even with spammers who are unreliable workers that make decisions at random, as compared to the trivial majority voting strategy. In [7], a sequential questioning strategy is developed to accommodate and recover from errors made by the workers. In many other works [8]- [10], spammers are filtered out when assigning tasks.…”

An Unsupervised Bayesian Neural Network for Truth Discovery in Social Networks

“…I T is common for agents in a social network to report conflicting opinions [1], [2], [3], [4], [5], [6], [7], [8]. Some of the agents are unreliable and maybe biased.…”