Addressing out-of-distribution label noise in webly-labelled data

Abstract

A recurring focus of the deep learning community is towards reducing the labeling effort. Data gathering and annotation using a search engine is a simple alternative to generating a fully human-annotated and human-gathered dataset. Although web crawling is very time efficient, some of the retrieved images are unavoidably noisy, i.e. incorrectly labeled. Designing robust algorithms for training on noisy data gathered from the web is an important research perspective that would render the building of datasets easier. In this paper we conduct a study to understand the type of label noise to expect when building a dataset using a search engine. We review the current limitations of stateof-the-art methods for dealing with noisy labels for image classification tasks in the case of web noise distribution. We propose a simple solution to bridge the gap with a fully clean dataset using Dynamic Softening of Out-of-distribution Samples (DSOS), which we design on corrupted versions of the CIFAR-100 dataset, and compare against state-of-the-art algorithms on the web noise perturbated MiniImageNet and Stanford datasets and on real label noise datasets: WebVision 1.0 and Clothing1M. Our work is fully reproducible https://git.io/JKGcj.

Review–A Survey of Learning from Noisy Labels

“…Up to now, very few studies address this issue, no efficient and effective solution can handle it either. 61.91 Co-teaching+ 47 81.61 JoCoR 34 77.94 Jo-SRC 50 86.66 Co-learning 49 87.57 DSOS 39 87.70 We think in the future study on label learning, it might be wise to focus more on these hard samples. We define "hard data" as data that distribute close to the decision boundary.…”

“…Then, it applies semi-supervised learning for the noisy subset without using the given noisy labels. DSOS 39 uses the entropy of the interpolation of prediction and given label to distinguish clean, in-distribution (ID) noise and out-of-distribution (OOD) noise. Then, it corrects the labels for ID samples and proposes a dynamic softening strategy for OOD samples to lower the harm of noisy labels.…”

Embedding contrastive unsupervised features to cluster in- and out-of-distribution noise in corrupted image datasets

“…Webvision [24] is a dataset constituted of 2.4 million images gathered using search queries on the same 1k classes as the ILSVRC12 [20] challenge. Albert et al [1] have shown that the noise present in Webvision is predominantly outof-distribution. Clothing1M [44] is a 1M images clothes classification dataset, popular dataset in the label noise community which, as shown by the authors, only contains ID noise.…”

“…Another class of noise robust algorithms have emerged recently to tackle noisy datasets presenting both in-and out-of-distribution noise. EvidentialMix [32] and DSOS [1] differentiates between ID noisy and OOD data using a custom noise landscape and JoSRC [46] evidences OOD samples as having a low agreement between two consistent views of the same image. The methods we compare against are described in more details in 4.1 and we direct the interested reader to a recent label noise survey by Song et al [15] for an in-depth overview of state-of-the-art label noise robust algorithms.…”

“…CNNs have been shown to completely overfit noisy samples in a dataset without proper regularization [48], which degrades performance. More specifically, the noise observed in web-crawled datasets has been categorized as both in-distribution and out-of-distribution, the latter being the dominant type [1]. While in-distribution (ID) noisy images can be directly used to train the network after correcting their assigned label, out-of-distribution (OOD) images cannot be assigned to any category.…”