On the Cost Complexity of Crowdsourcing

Fang, Yili; Sun, Hailong; Chen, Pengpeng; Huai, Jinpeng

doi:10.24963/ijcai.2018/212

“…We build our algorithm and analysis around this noise model but show that our analysis can be adapted to handle the case where the noise rates are conditioned on class membership. These noise models have been extensively studied in crowdsourcing literature [4,7,11,13,18,21] and are usually attributed to Dawid and Skene [5].…”

Section: Previous Workmentioning

confidence: 99%

“…PAC learning in crowdsourcing. Feng et al [7], in very recent work, develop PAC-style bounds for the cost complexity of learning an aggregation function that fits a crowd of workers with varying reliabilities. They focus on using PAC learning to train an aggregation function for the workers' labels; we, however, focus on using PAC learning to train a classifier that generalizes from worker labels.…”

Section: Previous Workmentioning

confidence: 99%

Crowdsourced PAC Learning under Classification Noise

Heinecke¹,

Reyzin²

2019

Preprint

View full text Add to dashboard Cite

In this paper, we analyze PAC learnability from labels produced by crowdsourcing. In our setting, unlabeled examples are drawn from a distribution and labels are crowdsourced from workers who operate under classification noise, each with their own noise parameter. We develop an end-to-end crowdsourced PAC learning algorithm that takes unlabeled data points as input and outputs a trained classifier. Our three-step algorithm incorporates majority voting, pure-exploration bandits, and noisy-PAC learning. We prove several guarantees on the number of tasks labeled by workers for PAC learning in this setting and show that our algorithm improves upon the baseline by reducing the total number of tasks given to workers. We demonstrate the robustness of our algorithm by exploring its application to additional realistic crowdsourcing settings.

show abstract

“…We build our algorithm and analysis around this noise model but show that our analysis can be adapted to handle the case where the noise rates are conditioned on class membership. These noise models have been extensively studied in crowdsourcing literature (Cao et al 2015;Fang et al 2018; Kang and Tay 2018;Li, Yu, and Zhou 2013;Wang and Zhou 2015;Zhou, Chen, and Li 2014) and are usually attributed to Dawid and Skene (1979).…”

Section: Previous Workmentioning

confidence: 99%

Crowdsourced PAC Learning under Classification Noise

Heinecke

¹

,

Reyzin

²

2019

HCOMP

5

0

View full text Add to dashboard Cite

In this paper, we analyze PAC learnability from labels produced by crowdsourcing. In our setting, unlabeled examples are drawn from a distribution and labels are crowdsourced from workers who operate under classification noise, each with their own noise parameter. We develop an end-to-end crowdsourced PAC learning algorithm that takes unlabeled data points as input and outputs a trained classifier. Our three-step algorithm incorporates majority voting, pure-exploration bandits, and noisy-PAC learning. We prove several guarantees on the number of tasks labeled by workers for PAC learning in this setting and show that our algorithm improves upon the baseline by reducing the total number of tasks given to workers. We demonstrate the robustness of our algorithm by exploring its application to additional realistic crowdsourcing settings.

show abstract

“…A significant prerequisite for the use of supervised learning is the availability of large, well-labeled datasets. Crowdsourcing (Zheng et al 2017;Fang et al 2018;Tong et al 2020) offers an affordable method of annotating data by using freelance workers located on the internet platforms such as Amazon Mechanical Turk 1 (AMT). When it comes to crowdsourced labeled data, a general rule of thumb is that the annotations may often be noisy because of the unskilled or malevolent behavior of workers.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Learning from Crowds

Chen

¹

,

Sun

²

,

Yang

³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Learning from Crowds (LFC) seeks to induce a high-quality classifier from training instances, which are linked to a range of possible noisy annotations from crowdsourcing workers under their various levels of skills and their own preconditions. Recent studies on LFC focus on designing new methods to improve the performance of the classifier trained from crowdsourced labeled data. To this day, however, there remain under-explored security aspects of LFC systems. In this work, we seek to bridge this gap. We first show that LFC models are vulnerable to adversarial examples---small changes to input data can cause classifiers to make prediction mistakes. Second, we propose an approach, A-LFC for training a robust classifier from crowdsourced labeled data. Our empirical results on three real-world datasets show that the proposed approach can substantially improve the performance of the trained classifier even with the existence of adversarial examples. On average, A-LFC has 10.05% and 11.34% higher test robustness than the state-of-the-art in the white-box and black-box attack settings, respectively.

show abstract

On the Cost Complexity of Crowdsourcing

Cited by 7 publications

References 13 publications

Crowdsourced PAC Learning under Classification Noise

Crowdsourced PAC Learning under Classification Noise

Crowdsourced PAC Learning under Classification Noise

Adversarial Learning from Crowds

Contact Info

Product

Resources

About