Collusion-Proof Result Inference in Crowdsourcing

Chen, Peng-Peng; Sun, Hailong; Fang, Yili; Huai, Jinpeng

doi:10.1007/s11390-018-1823-6

Cited by 14 publications

(7 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We set the threshold for ab j as 0.5. The quality of the aggregated answers declines than the answer from the individual worker when ab j is less than 0.5 [12].…”

Section: Updating the Parametersmentioning

confidence: 93%

“…1: Illustrating adversarial attacks in crowdsourcing systems, which comprises the main components, processes, and the attacking scenario in [11]. Moreover, the performance of the workers is closely related to the set of submissions of the workers [12]. We observe that incorporating these features and an iterative task allocation phase improves the performance of the system.…”

Section: Malicious Workers Normal Workersmentioning

confidence: 97%

See 1 more Smart Citation

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

First,

Second

2023

Preprint

View full text Add to dashboard Cite

Crowdsourcing is adopted as a fast and cost-effective system for human computation and acquiring data for training models in machine learning. Although crowdsourcing has broad applicability, it still has the following challenges. Normally, workers have access to crowdsourcing platforms with simple authentication mechanisms. As a result, malicious workers may get into the system and submit unreliable answers rendering the platform fraudulent. Moreover, when workers perform tasks, their lack of expertise and the difficulty of tasks affects the accuracy. Truth inference benefits the data quality and worker reliability. A truth inference algorithm estimates workers' trustworthiness and correctness of the answers from their responses. Besides, it helps in filtering out low-quality answers. However, due to adversarial attacks by malicious workers, ground truth inference algorithms perform poorly. This research considers how to defend against adversarial attacks on crowdsourcing platforms and improve the truth inference process. The proposed method estimates the trust and reliability scores of the workers and classifies them as normal and malicious workers. Based on this classification , tasks are assigned to the workers. Moreover, these predicted scores are used for inferring the correct answers, thereby improving the ground truth inference. Experiments consistently show that the proposed truth inference method is tolerant to adversarial attacks with competent accuracy.

show abstract

“…We set the threshold for ab j as 0.5. The quality of the aggregated answers declines than the answer from the individual worker when ab j is less than 0.5 [12].…”

Section: Updating the Parametersmentioning

confidence: 93%

Section: Malicious Workers Normal Workersmentioning

confidence: 97%

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

First,

Second

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Answer Aggregation in Crowdsourcing. In two-stage approaches, the true labels are inferred from the crowd labels using answer aggregation methods (Chen et al 2018(Chen et al , 2020a. Then, it applies the general supervised learning methods along with the inferred labels (Wang and Zhou 2016).…”

Section: Related Workmentioning

confidence: 99%

Adversarial Learning from Crowds

Chen

Sun

Yang

et al. 2022

AAAI

View full text Add to dashboard Cite

Learning from Crowds (LFC) seeks to induce a high-quality classifier from training instances, which are linked to a range of possible noisy annotations from crowdsourcing workers under their various levels of skills and their own preconditions. Recent studies on LFC focus on designing new methods to improve the performance of the classifier trained from crowdsourced labeled data. To this day, however, there remain under-explored security aspects of LFC systems. In this work, we seek to bridge this gap. We first show that LFC models are vulnerable to adversarial examples---small changes to input data can cause classifiers to make prediction mistakes. Second, we propose an approach, A-LFC for training a robust classifier from crowdsourced labeled data. Our empirical results on three real-world datasets show that the proposed approach can substantially improve the performance of the trained classifier even with the existence of adversarial examples. On average, A-LFC has 10.05% and 11.34% higher test robustness than the state-of-the-art in the white-box and black-box attack settings, respectively.

show abstract

“…, K}. The colluding workers are more malicious, and they collude and try to guide the outcome of the majority vote [7], [8], [9]. In our experiment, a label is sampled from a uniform distribution over {1, .…”

Section: Robustness Against Harmful Workersmentioning

confidence: 99%

“…This is attributed to the significant variations in the abilities and motivations of the human workers, as well as the anonymity of crowdsourcing workers. In addition, there are spam workers who provide random answers without looking at the tasks [5], [6], as well as colluding workers who share their answers with other workers [7], [8], [9], which also contribute to the variability in the reliability. To reduce the impact of incorrect responses, several studies have attempted to improve the quality by aggregating multiple redundantly collected responses from different workers.…”

Section: Introductionmentioning

confidence: 99%

Mitigating Observation Biases in Crowdsourced Label Aggregation

Ueda

Takeuchi

Kashima

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Crowdsourcing has been widely used to efficiently obtain labeled datasets for supervised learning from large numbers of human resources at low cost. However, one of the technical challenges in obtaining high-quality results from crowdsourcing is dealing with the variability and bias caused by the fact that it is humans execute the work, and various studies have addressed this issue to improve the quality by integrating redundantly collected responses. In this study, we focus on the observation bias in crowdsourcing. Variations in the frequency of worker responses and the complexity of tasks occur, which may affect the aggregation results when they are correlated with the quality of the responses. We also propose statistical aggregation methods for crowdsourcing responses that are combined with an observational data bias removal method used in causal inference. Through experiments using both synthetic and real datasets with/without artificially injected spam and colluding workers, we verify that the proposed method improves the aggregation accuracy in the presence of strong observation biases and robustness to both spam and colluding workers.

show abstract

Collusion-Proof Result Inference in Crowdsourcing

Cited by 14 publications

References 31 publications

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

Adversarial Learning from Crowds

Mitigating Observation Biases in Crowdsourced Label Aggregation

Contact Info

Product

Resources

About