Multi-Object Classification via Crowdsourcing With a Reject Option

Abstract

Abstract-Consider designing an effective crowdsourcing system for an M -ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result. We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond. For example, in mismatched speech transcription, workers who do not know the language may not be able to respond to microtasks focused on phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.

Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

“…In this paper, we extend our work [18,19] by further taking the spammers' effect on the system into consideration. We study the scenario where spammers also exist in the crowd, who participate in the task only to earn some free money without regard to the quality of their answers.…”

“…In our previous work [18,19], we proposed a novel weighted majority voting method for crowdsourced classification, which was derived by solving the following optimization problem…”

“…As an example, in mismatched speech transcription, i.e., transcription by crowd workers who do not know the language, workers may not be able to perceive the phonological dimensions they are tasked to differentiate [17]. We investigated the optimal aggregation rule where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond [18,19].…”

Decision Tree Design for Classification in Crowdsourcing Systems

“…In recent work on classification in crowdsourcing systems, complex questions are often replaced by a set of simpler binary questions (microtasks) to enhance classification performance [1]- [4]. This is especially helpful in situations where crowd workers lack expertise for responding to complex questions directly.…”

“…These binary questions can be posted as "microtasks" on crowdsourcing platforms like Amazon Mechanical Turk [5]. To improve classification performance in crowdsourcing systems, most of the works in the literature focus on enhancing the quality of individual tests, by designing fusion rules to combine decisions from heterogeneous workers [1]- [4], [6], [7], and by investigating the assignment of different tests to different workers depending upon their skill level [8], [9]. These problems have also been extended to budget-constrained environments to improve classification performance [10]- [12].…”