Deep Neural Networks for Corrupted Labels

Jindal, Ishan; Nokleby, Matthew; Pressel, Daniel; Chen, Xuewen; Singh, Harpreet

doi:10.1007/978-3-030-31756-0_7

Cited by 45 publications

(78 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There have been investigations [37]- [39] towards the learning of discriminative models using noisy labeled datasets. By the small-loss approach, which updates the network with only the samples that have a small loss [40], it is possible to reduce the impact of noise samples in DANN [16] for UDA. Another strategy [41]- [43] is to detect noisy annotations by using multiple networks to create different viewpoints of each sample and these methods have produced better outcomes.…”

Section: Methodsmentioning

confidence: 99%

Divergence Optimization for Noisy Universal Domain Adaptation

Hashimoto

Ushiku

2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

To transfer the knowledge learned from a labeled source domain to an unlabeled target domain, many studies have worked on universal domain adaptation (UniDA), where there is no constraint on the label sets of the source domain and target domain. However, the existing UniDA methods rely on source samples with correct annotations. Due to the limited resources in the real world, it is difficult to obtain a large amount of perfectly clean labeled data in a source domain in some applications. As a result, we propose a novel realistic scenario named Noisy UniDA, in which classifiers are trained using noisy labeled data from the source domain as well as unlabeled domain data from the target domain that has an uncertain class distribution. A multi-head convolutional neural network framework is proposed in this paper to address all of the challenges faced in the Noisy UniDA at once. Our network comprises a single common feature generator and multiple classifiers with various decision bounds. We can detect noisy samples in the source domain, identify unknown classes in the target domain, and align the distribution of the source and target domains by optimizing the divergence between the outputs of the various classifiers. The proposed method outperformed the existing methods in most of the settings after a thorough analysis of the various domain adaption scenarios. The source code is available at https://github.com/YU1ut/Divergence-Optimization.

show abstract

Section: Methodsmentioning

confidence: 99%

Divergence Optimization for Noisy Universal Domain Adaptation

Hashimoto

Ushiku

2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Learning from noisy labels is an important and well-studied problem with a vast literature (Bar et al, 2021;Frénay & Verleysen, 2013;Gamberger et al, 1999;Jiang et al, 2018;Kumar & Amid, 2021;Liu & Tao, 2015;Majidi et al, 2021;Natarajan et al, 2013;Pleiss et al, 2020;Ren et al, 2018) -see (Song et al, 2022) for a recent survey. The fundamental difference between our setting and papers in this literature is that the noise introduced by the teacher is structured, and this is a crucial observation we utilize in our design.…”

Section: Learning With Noisy Data Techniquesmentioning

confidence: 99%

SLaM: Student-Label Mixing for Semi-Supervised Knowledge Distillation

Kontonis¹,

Iliopoulos²,

Trinh³

et al. 2023

Preprint

View full text Add to dashboard Cite

Semi-supervised knowledge distillation is a powerful training paradigm for generating compact and lightweight student models in settings where the amount of labeled data is limited but one has access to a large pool of unlabeled data. The idea is that a large teacher model is utilized to generate "smoothed" pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success in a wide variety of applications, a shortcoming of this approach is that the teacher's pseudo-labels are often noisy, leading to impaired student performance. In this paper, we present a principled method for semi-supervised knowledge distillation that we call Student-Label Mixing (SLaM) and we show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. Finally, we show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, and provide the first convergence analysis for so-called "forward lossadjustment" methods.

show abstract

“…This line of works regulate the importance of each sample in the parameter estimation to reduce the noise effect [11,[28][29][30][31]. It can be implemented in the perspectives of the label re-weighting or the sample pair re-weighting based on the modeling.…”

Section: Learning Via Sample Re-weightingmentioning

confidence: 99%

“…Wang et al, [33] utilized the local intrinsic dimensionality to dynamically adapt the weight between the label and the prediction for Bootstrapping [10]. Jiang et al, [29,30] designed a curriculum to measure the quality of each sample, and added the learned weights into the loss function to reduce the label noise effect. Shu et al, [34] followed the similar motivation but constructed an explicit mapping to reweight the training samples under label noise.…”

Section: Learning Via Sample Re-weightingmentioning

confidence: 99%

Latent Class-Conditional Noise Model

Yao¹,

Han²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

Learning with noisy labels has become imperative in the Big Data era, which saves expensive human labors on accurate annotations. Previous noise-transition-based methods have achieved theoretically-grounded performance under the Class-Conditional Noise model (CCN). However, these approaches builds upon an ideal but impractical anchor set available to pre-estimate the noise transition. Even though subsequent works adapt the estimation as a neural layer, the ill-posed stochastic learning of its parameters in back-propagation easily falls into undesired local minimums. We solve this problem by introducing a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. By projecting the noise transition into the Dirichlet space, the learning is constrained on a simplex characterized by the complete dataset, instead of some ad-hoc parametric space wrapped by the neural layer. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels to train the classifier and to model the noise. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples. We further generalize LCCN to different counterparts compatible with open-set noisy labels, semi-supervised learning as well as cross-model training. A range of experiments demonstrate the advantages of LCCN and its variants over the current state-of-the-art methods. The code is available at here.

show abstract

Deep Neural Networks for Corrupted Labels

Cited by 45 publications

References 28 publications

Divergence Optimization for Noisy Universal Domain Adaptation

Divergence Optimization for Noisy Universal Domain Adaptation

SLaM: Student-Label Mixing for Semi-Supervised Knowledge Distillation

Latent Class-Conditional Noise Model

Contact Info

Product

Resources

About