Towards Robust Learning with Different Label Noise Distributions

Abstract

Noisy labels are an unavoidable consequence of automatic image labeling processes to reduce human supervision. Training in these conditions leads Convolutional Neural Networks to memorize label noise and degrade performance. Noisy labels are therefore dispensable, while image content can be exploited in a semi-supervised learning (SSL) setup. Handling label noise then becomes a label noise detection task. Noisy/clean samples are usually identified using the small loss trick, which is based on the observation that clean samples represent easier patterns and, therefore, exhibit a lower loss. However, we show that different noise distributions make the application of this trick less straightforward. We propose to continuously relabel all images to reveal a loss that facilitates the use of the small loss trick with different noise distributions. SSL is then applied twice, once to improve the clean-noisy detection and again for training the final model. We design an experimental setup for better understanding the consequences of differing label noise distributions and find that non-uniform out-of-distribution noise better resembles real-world noise. We show that SSL outperforms other alternatives when using oracles and demonstrate substantial improvements across five datasets of our label noise Distribution Robust Pseudo-Labeling (DRPL). We further study the effects of label noise memorization via linear probes and find that in most cases intermediate features are not affected by label noise corruption. Code and details to reproduce our framework will be made available.

When Do Curricula Work?

“…For a short time learning, the two could be similar. Among much work on minimizing large current loss (i.e., Fan et al (2017); Zhang et al (2019a); Jiang et al (2019); Ortego et al (2019) and reference therein), Kawaguchi & Lu (2020) introduced ordered stochastic gradient descent (SGD), which purposely biased toward those instances with higher current losses. They empirically show that the ordered SGD outperforms the standard SGD (Bottou et al, 2018) when decreasing the learning rate, even though it is not better initially.…”

PropMix: Hard Sample Filtering and Proportional MixUp for Learning with Noisy Labels

“…SSL methods [30,42] first classify samples as clean or noisy, where the noisy samples are re-labelled by the model, and these clean and noisy sets are combined with MixMatch [4]. As mentioned before, SSL methods have two issues: 1) the training set size for the MixMatch stage is limited by the clean set size that reduces with increasing label noise, and 2) the noisy sample re-labelling accuracy also reduces with increasing label noise.…”

“…Moreover, in high noise rate scenarios, the filtered clean set can be too small to train the model. The noisy set can be used after re-labelling the noisy samples [30,42,50], which works well for low noise rate problems, but as the noise rate increases, this approach is not effective given that the incorrectly re-labelled noisy samples tend to bias the training. The use of clean validation sets in a meta-learning approach [2] can reduce this issue, but the existence of a clean validation set may be infeasible in real world applications.…”

“…A common approach to address this challenge is based on semi-supervised learning (SSL) methods [10,30,42,43], based on a 2-stage process formed by an unsupervised learning method to classify training samples as clean or noisy, followed by SSL that "Mix-Matches" [4] the labelled set formed by the samples classified as clean, and the re-labelled samples from the noisy set. There are two issues with such approach.…”

Reliable Label Bootstrapping for Semi-Supervised Learning

“…Automatic annotation of data becomes a plausible answer [16] that unavoidably infers some incorrect or noisy labels. To prevent harming the representations learned [17], label noise-resistant training of CNNs is often necessary [18], [19], [20], [21]. In particular, the small loss trick [17] associates examples with a low (high) training loss to samples with clean (noisy) labels.…”