Worker ranking determination in crowdsourcing platforms using aggregation functions

Sanchez-Charles, David; Nin, Jordi; Solé, Marc; Muntés-Mulero, Víctor

doi:10.1109/fuzz-ieee.2014.6891807

Cited by 30 publications

(21 citation statements)

References 19 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Relevant workers for tasks are filtered using their profiles information. [34,43]. Based on the level of task quality delivered, these profiles are updated [38].…”

Section: Crowd Selection On the Basis Of Individual Profilesmentioning

confidence: 99%

“…Organization selects skilled work force for various tasks [42]. As the quality depends highly on workers' skills [43], an Initial screening of the crowd workers is carried out [50]. ese screening are also referred as skill assessment which evaluates the crowd according to possessing skills and they are helpful in matching skilled labor to a task [26,51].…”

Section: Crowd Selection On the Basis Of Skillsmentioning

confidence: 99%

See 1 more Smart Citation

Multicriteria‐Based Crowd Selection Using Ant Colony Optimization

et al. 2021

View full text Add to dashboard Cite

Internet-enabled technologies have provided a way for people to communicate and collaborate with each other. The collaboration and communication made crowdsourcing an efficient and effective activity. Crowdsourcing is a modern paradigm that employs cheap labors (crowd) for accomplishing different types of tasks. The task is usually posted online as an open call, and members of the crowd self-select a task to be carried out. Crowdsourcing involves initiators or crowdsourcers (an entity usually a person or an organization who initiate the crowdsourcing process and seek out the ability of crowd for a task), the crowd (online participant who is a having a particular background, qualification, and experience for accomplishing task in crowdsourcing activity), crowdsourcing task (the activity in which the crowd contribute), the process (how the activity is carried out), and the crowdsourcing platform (software or market place) where requesters offer various tasks and crowd workers complete these tasks. As the crowdsourcing is carried out in the online environment, it gives rise to certain challenges. The major problem is the selection of crowd that is becoming a challenging issue with the growth in crowdsourcing popularity. Crowd selection has been significantly investigated in crowdsourcing processes. Nonetheless, it has observed that the selection is based only on a single feature of the crowd worker which was not sufficient for appropriate crowd selection. For addressing the problem of crowd selection, a novel “ant colony optimization-based crowd selection method” (ACO-CS) is presented in this paper that selects a crowd worker based on multicriteria features. By utilizing the proposed model, the efficiency and effectiveness of crowdsourcing activity will be increased.

show abstract

“…Relevant workers for tasks are filtered using their profiles information. [34,43]. Based on the level of task quality delivered, these profiles are updated [38].…”

Section: Crowd Selection On the Basis Of Individual Profilesmentioning

confidence: 99%

Section: Crowd Selection On the Basis Of Skillsmentioning

confidence: 99%

Multicriteria‐Based Crowd Selection Using Ant Colony Optimization

et al. 2021

View full text Add to dashboard Cite

show abstract

“…A classification problem with crowdsourcing, where taxonomy and dichotomous keys are used to design binary questions, is considered in [9]. New aggregation rules that mitigate the unreliability of the crowd and improve the crowdsourcing system performance are investigated in [10,11]. In our research group, we employed binary questions and studied the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification [9,12].…”

Section: Introductionmentioning

confidence: 99%

Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

Varshney

2018

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We explore the design of an effective crowdsourcing system for an M -ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance.

show abstract

“…Without the reject option, noisy responses to tasks cannot be tagged before aggregation so appropriate weights cannot be assigned [37]. For instance, the popular majority voting rule weights all answers equally [42], though new weighted aggregation rules have also been developed [37], [39]. We employed error-control codes and decoding algorithms to design reliable crowdsourcing systems with unreliable workers [22].…”

Section: Introductionmentioning

confidence: 99%

Multi-Object Classification via Crowdsourcing With a Reject Option

Vempaty

Varshney

et al. 2017

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-Consider designing an effective crowdsourcing system for an M -ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.

show abstract

Worker ranking determination in crowdsourcing platforms using aggregation functions

Cited by 30 publications

References 19 publications

Multicriteria‐Based Crowd Selection Using Ant Colony Optimization

Multicriteria‐Based Crowd Selection Using Ant Colony Optimization

Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

Multi-Object Classification via Crowdsourcing With a Reject Option

Contact Info

Product

Resources

About