Semi-Supervised Multi-Label Learning from Crowds via Deep Sequential Generative Model

Shi, Wanli; Sheng, Victor S.; Li, Xiang; Gu, Bin

doi:10.1145/3394486.3403167

Cited by 9 publications

(11 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We note that Refs. [21][22][23][24][25] also proposed deep generative crowdsourcing learning approaches on the basis of variational autoencoders and their extended models. For example, Ref.…”

Section: Related Workmentioning

confidence: 99%

“…[21] minimized the reconstruction errors in crowdsourcing annotation with a variational autoencoder [18] ; and Refs. [22][23][24] proposed semi-supervised crowdsourcing classification and clustering learning methods using unlabeled data based on semi-supervised variational autoencoders [19] . Ref.…”

Section: Related Workmentioning

confidence: 99%

“…Most of the works in this regard are based on the variational autoencoder model and its improvements [18][19][20] . Works on the same topic as this study are still relatively scarce [21][22][23][24][25] and ignore the use of correlations among workers. Nevertheless, available works on non-deep crowdsourcing [10][11][12] show that worker correlation modeling and utilization can help to improve the crowdsourcing learning effect.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

Li¹,

Wei²,

Huang³

2022

International Journal of Software and Informatics

View full text Add to dashboard Cite

Traditional supervised learning requires the groundtruth labels for the training data, which can be difficult to collect in many cases. In contrast, crowdsourcing learning collects noisy annotations from multiple non-expert workers and infers the latent true labels through some aggregation approach. In this paper, we notice that existing deep crowdsourcing work does not sufficiently model worker correlations, which is, however, shown to be helpful for learning by previous non-deep learning approaches. We propose a deep generative crowdsourcing learning approach to incorporate the strengths of Deep Neural Networks (DNNs) and exploit worker correlations. The model comprises a DNN classifier as a prior and an annotation generation process. A mixture model of workers' capabilities within each class is introduced into the annotation generation process for worker correlation modeling. For adaptive trade-off between model complexity and data fitting, we implement fully Bayesian inference. Based on the naturalgradient stochastic variational inference techniques developed for the Structured Variational AutoEncoder (SVAE), we combine variational message passing for conjugate parameters and stochastic gradient descent for DNN parameters into a unified framework for efficient endto-end optimization. Experimental results on 22 real crowdsourcing datasets demonstrate the effectiveness of the proposed approach.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

Li¹,

Wei²,

Huang³

2022

International Journal of Software and Informatics

View full text Add to dashboard Cite

show abstract

“…An intriguing line of investigation by Goldberger examined the problem of inexact labels [31]. More recently, the success of deep neural networks has prompted work on deep generative models [32]- [34]. Regardless of the specific approach, nearly all work in this area attempts to model either the parameters of annotators' reliabilities, or the confusion matrices associated with each annotator.…”

Section: Related Workmentioning

confidence: 99%

“…The generative model proposed by Welinder et al is highly parameterized and considerably more complex than the model suggested by Whitehill et al Later models build on this idea, such as that of Atarashi et al, add even more parameterization in the form of latent variables [33]. Latent variable models are intractable for EM algorithms; more recent authors have turned to deep neural networks for approximating these more complex environments [32]- [34]. Each of these models, however, shares lineage with the work of Welinder et al, and the general class of multidimensional label generation dynamics is well-represented by their work.…”

Section: B Multidimensional Parameterization Of Label Generationmentioning

confidence: 99%

OpinionRank: Extracting Ground Truth Labels from Unreliable Expert Opinions with Graph-Based Spectral Ranking

Dawson¹,

Polikar²

2021

Preprint

View full text Add to dashboard Cite

As larger and more comprehensive datasets become standard in contemporary machine learning, it becomes increasingly more difficult to obtain reliable, trustworthy label information with which to train sophisticated models. To address this problem, crowdsourcing has emerged as a popular, inexpensive, and efficient data mining solution for performing distributed label collection. However, crowdsourced annotations are inherently untrustworthy, as the labels are provided by anonymous volunteers who may have varying, unreliable expertise. Worse yet, some participants on commonly used platforms such as Amazon Mechanical Turk may be adversarial, and provide intentionally incorrect label information without the end user's knowledge. We discuss three conventional models of the label generation process, describing their parameterizations and the model-based approaches used to solve them. We then propose OpinionRank, a model-free, interpretable, graph-based spectral algorithm for integrating crowdsourced annotations into reliable labels for performing supervised or semi-supervised learning. Our experiments show that OpinionRank performs favorably when compared against more highly parameterized algorithms. We also show that OpinionRank is scalable to very large datasets and numbers of label sources, and requires considerably less computational resources than previous approaches.

show abstract

Bayesian Crowdsourcing with Constraints

Traganitis

Giannakis

2021

Machine Learning and Knowledge Discovery in Databases. Research Track

View full text Add to dashboard Cite

Semi-Supervised Multi-Label Learning from Crowds via Deep Sequential Generative Model

Cited by 9 publications

References 28 publications

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

OpinionRank: Extracting Ground Truth Labels from Unreliable Expert Opinions with Graph-Based Spectral Ranking

Bayesian Crowdsourcing with Constraints

Contact Info

Product

Resources

About