Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?

Abstract

We explore the design of an e ective crowdsourcing system for an M-ary classi cation task. Crowd workers complete simple binary microtasks whose results are aggregated to give the nal classi cation decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. Additionally, the workers report quantized con dence levels when they are able to submit de nitive answers. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classi cation performance. We obtain a couterintuitive result that the classi cation performance does not bene t from workers reporting quantized con dence. erefore, the crowdsourcing system designer should employ the reject option without requiring con dence reporting. CCS CONCEPTS•Human-centered computing →Social network analysis;

Optimal Crowdsourced Classification with a Reject Option in the Presence of Spammers

“…In our previous work [18,19], we proposed a novel weighted majority voting method for crowdsourced classification, which was derived by solving the following optimization problem…”

“…In this paper, we extend our work [18,19] by further taking the spammers' effect on the system into consideration. We study the scenario where spammers also exist in the crowd, who participate in the task only to earn some free money without regard to the quality of their answers.…”

“…As an example, in mismatched speech transcription, i.e., transcription by crowd workers who do not know the language, workers may not be able to perceive the phonological dimensions they are tasked to differentiate [17]. We investigated the optimal aggregation rule where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond [18,19].…”

Prospect Theory Based Crowdsourcing for Classification in the Presence of Spammers

“…C ROWDSOURCING has attracted intense interest in recent years as a new paradigm for distributed inference. It harnesses the intelligence of the crowd, by exploiting the inexpensive and online labor markets in an effective manner [1]- [7]. Crowdsourcing enables a new framework to utilize distributed human wisdom to solve problems that machines cannot perform well, like handwriting recognition, anomaly detection, voice transcription, and image labelling [8]- [11].…”

Decision Tree Design for Classification in Crowdsourcing Systems

“…In recent work on classification in crowdsourcing systems, complex questions are often replaced by a set of simpler binary questions (microtasks) to enhance classification performance [1]- [4]. This is especially helpful in situations where crowd workers lack expertise for responding to complex questions directly.…”