Task selection for bandit-based task assignment in heterogeneous crowdsourcing

Abstract

Task selection (picking an appropriate labeling task) and worker selection (assigning the labeling task to a suitable worker) are two major challenges in task assignment for crowdsourcing. Recently, worker selection has been successfully addressed by the bandit-based task assignment (BBTA) method, while task selection has not been thoroughly investigated yet. In this paper, we experimentally compare several task selection strategies borrowed from active learning literature, and show that the least confidence strategy significantly improves the performance of task assignment in crowdsourcing.

MCDM Approach for Assigning Task to the Workers by Selected Features Based on Multiple Criteria in Crowdsourcing

“…ese separation measures are calculated by using equations ( 10) and (11). Ideal separation measures (S + ) for W1, W2, Ranking is done upon the value of P i , and the high value of P i shows the top alternative.…”

“…e goal is to select the worker that is suitable and good for the task [11]. A dynamic solution is also presented for the assignment problem of tasks in the crowdsourcing platforms.…”

Crowdsourcing usage, task assignment methods, and crowdsourcing platforms: A systematic literature review

“…19 The details of some of the papers are given in Table 11. 20 Task recommendation in crowdsourcing systems 0.5 0 1 1 2.5 Mao et al 40 Pricing CSD tasks 0.5 1 1 1 3.5 Stol and Fitzgerald 41 CSD case study 1 0 0.5 0.5 2 Cheung et al 42 Distributed time-sensitive task selection 0.5 0 1 0.5 2 Saremi and Yang 18 Simulation of workers and task completion 0 1 0 1 2 Maalej and Ellmann 7 On the similarity of task contexts 0 1 0 0 1 Roy et al 24 Task assignment optimization 0.5 0 1 1 2.5 Qiu et al 26 Crowdsourcing tasks accuracy through behavior prediction and user selection 0.5 0.5 1 1 3 Alelyani and Yang 19 Software crowdsourcing and developers behavior study 0.5 1 1 1 3.5 Machado et al 4 Task allocation in crowdsourcing 0.5 1 1 0.5 3 Borromeo et al 43 Crowd and task influence on crowdsourced work quality 0.5 0.5 0 1 2 Yang et al 12 Dynamic decision support for crowd workers 1 1 1 0.5 3.5 Han et al 44 Budgeted task scheduling for crowdsourced knowledge acquisition 0.5 0 0 0.5 1 Li et al 45 Crowdsourced data management overview and challenges 1 1 0.5 1 3.5 Cui et al 28 Learning task allocation strategies 1 1 1 1 4 Wang et al 14 Recommending CSD by considering Skill Improvement 1 1 0.5 0.5 3 Sanagavarapu and Reddy 46 Crowdsourcing security-opportunities and challenges 0 0.5 0 1 1.5 Costa and Nascimento 37 In-route task selection in crowdsourcing 0.5 0 1 0.5 2 Jiang et al 47 Crowdsourcing task assignment based on workers quality 0.5 0 1 1 2.5 Shin and Paek 48 Task classification in crowdsourcing 0 0 0 1 1 Simão Filho et al 17 Task allocation in distributed SD 0.5 1 1 0 2.5 Pan et al 34 Recommendation of crowdsourcing tasks based on Word2vec semantic tags 0.5 0 1 1 2.5 Weidema et al 49 Toward microtask crowdsourcing software design work 0 0 0 1 1 Song et al 50 Task selection and allocation framework 1 0 1 0 2 Li et al 23 Crowdsourced data management: a survey 1 1 0 1 3 Zhang and Sugiyama 25 Task selection in crowdsourcing 1 1 1 0.5 3.5 Zhang et al 51 On reliable task as...…”

“…Zhang and Sugiyama 25 Task selection in crowdsourcing Bandit-based task assignment is a bandit formulation for the assignment of tasks in heterogeneous crowdsourcing. Every bandit arm represents a worker.…”