Adversarial Learning from Crowds

Abstract

Learning from Crowds (LFC) seeks to induce a high-quality classifier from training instances, which are linked to a range of possible noisy annotations from crowdsourcing workers under their various levels of skills and their own preconditions. Recent studies on LFC focus on designing new methods to improve the performance of the classifier trained from crowdsourced labeled data. To this day, however, there remain under-explored security aspects of LFC systems. In this work, we seek to bridge this gap. We first show that LFC models are vulnerable to adversarial examples---small changes to input data can cause classifiers to make prediction mistakes. Second, we propose an approach, A-LFC for training a robust classifier from crowdsourced labeled data. Our empirical results on three real-world datasets show that the proposed approach can substantially improve the performance of the trained classifier even with the existence of adversarial examples. On average, A-LFC has 10.05% and 11.34% higher test robustness than the state-of-the-art in the white-box and black-box attack settings, respectively.

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

“…However, all these methods do not have the advantages of the recently proposed neuralized HMM-based graphical models [18,19] and our Neural-Hidden-CRF in principled modeling for variants of interest and in harnessing the context information that provided by advanced deep learning models. Additionally, it is worth mentioning the presence of numerous established WS methods that address the normal independent classification scenario [3,5,[43][44][45].…”

“…Essentially, our model given by Equation 7or Equation 9is inherently aligns with the underlying theory mentioned in Section 2.1. 3 The Embodiment of the Global Optimization Perspective. We can notice that the our pseudo-potential functions and feature functions in Equations 2-3 do not have a direct probabilistic interpretation, but instead represent constraints or scores on the configurations of the random variable.…”

“…This is because: (i) For the probability calculation complexity, since the "source-specific perturbation" methods require to learn 𝐽 source-specific models, their complexities (𝑂 (𝐽 𝐿𝐾 2 )) are 𝐽 times the corresponding complexity of CRF (𝑂 (𝐿𝐾 2 )). Also, when our method utilizes the dynamic programming algorithm to compute our Equation A.3 (the most significant consumers of computing) in the Appendix A.1, its complexity is also 𝑂 (𝐽 𝐿𝐾 2 ); (ii) For the inference complexity, each Viterbi decoding process required by these methods is the same as for the CRF, and therefore the complexities are all 𝑂 (𝐿𝐾 2 ) 3. Note that here we consider the complexities on one instance, and by the general convention, we do not consider the complexity arising from the deep neural backbone, which has an equivalent effect for all methods.…”