Learning from Noisy Large-Scale Datasets with Minimal Supervision

Veit, Andreas; Alldrin, Neil; Chechik, Gal; Krasin, Ivan; Gupta, Abhinav; Belongie, Serge

doi:10.1109/cvpr.2017.696

Cited by 420 publications

(270 citation statements)

References 22 publications

(22 reference statements)

Supporting

Mentioning

261

Contrasting

Order By: Relevance

“…16: FaceScrub with noise level of 30% not rely on clean labels to remove the noise. More accurate solutions, which rely on clean labels during the training phase have thus been explored (e.g., [16], [22], [41]). These solutions generally train a separate network for distinguishing noisy labels from clean ones.…”

Section: Related Workmentioning

confidence: 99%

Robust Anomaly Detection on Unreliable Data

Zhao

Cerf

Birke

et al. 2019

2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

View full text Add to dashboard Cite

Classification algorithms have been widely adopted to detect anomalies for various systems, e.g., IoT, cloud and face recognition, under the common assumption that the data source is clean, i.e., features and labels are correctly set. However, data collected from the wild can be unreliable due to careless annotations or malicious data transformation for incorrect anomaly detection. In this paper, we present a two-layer on-line learning framework for robust anomaly detection (RAD) in the presence of unreliable anomaly labels, where the first layer is to filter out the suspicious data, and the second layer detects the anomaly patterns from the remaining data. To adapt to the on-line nature of anomaly detection, we extend RAD with additional features of repetitively cleaning, conflicting opinions of classifiers, and oracle knowledge. We on-line learn from the incoming data streams and continuously cleanse the data, so as to adapt to the increasing learning capacity from the larger accumulated data set. Moreover, we explore the concept of oracle learning that provides additional information of true labels for difficult data points. We specifically focus on three use cases, (i) detecting 10 classes of IoT attacks, (ii) predicting 4 classes of task failures of big data jobs, (iii) recognising 20 celebrities faces. Our evaluation results show that RAD can robustly improve the accuracy of anomaly detection, to reach up to 98% for IoT device attacks (i.e., +11%), up to 84% for cloud task failures (i.e., +20%) under 40% noise, and up to 74% for face recognition (i.e., +28%) under 30% noisy labels. The proposed RAD is general and can be applied to different anomaly detection algorithms.

show abstract

Section: Related Workmentioning

confidence: 99%

Robust Anomaly Detection on Unreliable Data

Zhao

Cerf

Birke

et al. 2019

2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

View full text Add to dashboard Cite

show abstract

“…68.8 ± 0.9 69.8 ± 0.7 Lq,prune (n1 = 20|n1 = 15) 69.0 ± 0.6 70.2 ± 0.5 variants, that is, applying mixup to examples of the same batch after random permutation, or to examples of two different batches. No major differences are observed.…”

Section: Cce (Same As Baseline Inmentioning

confidence: 99%

“…However, such estimation is not trivial, and it assumes that the only possible type of noise is flipping labels. Other approaches use noise-robust loss functions to mitigate the effect of label noise [14], or leverage an additional set of curated data, for example to train a label cleaning network in order to reduce the noise of a dataset [15]. Conversely, learning with noisy labels has received little attention in sound recognition, probably given the traditional paradigm of learning from relatively small and exhaustively labeled (hence clean) datasets.…”

Section: Introductionmentioning

confidence: 99%

Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers

Fonseca

Font

Serra

2019

2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)

View full text Add to dashboard Cite

Label noise is emerging as a pressing issue in sound event classification. This arises as we move towards larger datasets that are difficult to annotate manually, but it is even more severe if datasets are collected automatically from online repositories, where labels are inferred through automated heuristics applied to the audio content or metadata. While learning from noisy labels has been an active area of research in computer vision, it has received little attention in sound event classification. Most recent computer vision approaches against label noise are relatively complex, requiring complex networks or extra data resources. In this work, we evaluate simple and efficient model-agnostic approaches to handling noisy labels when training sound event classifiers, namely label smoothing regularization, mixup and noise-robust loss functions. The main advantage of these methods is that they can be easily incorporated to existing deep learning pipelines without need for network modifications or extra resources. We report results from experiments conducted with the FSDnoisy18k dataset. We show that these simple methods can be effective in mitigating the effect of label noise, providing up to 2.5% of accuracy boost when incorporated to two different CNNs, while requiring minimal intervention and computational overhead.

show abstract

“…Most of the original methods for label adjustment adopt EM‐like algorithms to infer the true labels of training samples . Alternatively, noisy labels can also be adjusted by the predictions of deep models . The methods based on sample selection have obtained state‐of‐the‐art performance in addressing noisy labels .…”

Section: Introductionmentioning

confidence: 99%

“…[39][40][41] Alternatively, noisy labels can also be adjusted by the predictions of deep models. [42][43][44] The methods based on sample selection have obtained state-of-the-art performance in addressing noisy labels. 45,46 Specifically, Jiang et al proposed learning a data-driven curriculum that provides a sample weighting scheme for deep models trained on noisy labels.…”

Section: Introductionmentioning

confidence: 99%

Breast tumor classification through learning from noisy labeled ultrasound images

et al. 2019

View full text Add to dashboard Cite

Purpose To train deep learning models to differentiate benign and malignant breast tumors in ultrasound images, we need to collect many training samples with clear labels. In general, biopsy results can be used as benign/malignant labels. However, most clinical samples generally do not have biopsy results. Previous works have proposed generating benign/malignant labels according to Breast Imaging, Reporting and Data System (BI‐RADS) ratings. However, this approach will cause noisy labels, which means that the benign/malignant labels produced from BI‐RADS diagnoses may be inconsistent with the ground truths. Consequently, deep models will overfit the noisy labels and hence obtain poor generalization performance. In this work, we mainly focus on how to reduce the negative effect of noisy labels when they are used to train breast tumor classification models. Methods We propose an effective approach called noise filter network (NF‐Net) to address the problem of noisy labels when training breast tumor classification models. Specifically, to prevent deep models from overfitting the noisy labels, we propose incorporating two softmax layers for classification. Additionally, to strengthen the effect of clean labels, we design a teacher–student module for distilling the knowledge of clean labels. Results We conduct extensive comparisons with the existing works on addressing noisy labels. Our method achieves a classification accuracy of 73%, with a precision of 69%, recall of 80%, and F1‐score of 0.74. This result is significantly better than those of the existing state‐of‐the‐art works on addressing noisy labels. Conclusions This work provides a means to overcome the label shortage problem in training breast tumor classification models. Specifically, we can generate benign/malignant labels according to the BI‐RADS ratings. Although this approach will cause noisy labels, the design of NF‐Net can effectively reduce the negative effect of such labels.

show abstract

Learning from Noisy Large-Scale Datasets with Minimal Supervision

Cited by 420 publications

References 22 publications

Robust Anomaly Detection on Unreliable Data

Robust Anomaly Detection on Unreliable Data

Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers

Breast tumor classification through learning from noisy labeled ultrasound images

Contact Info

Product

Resources

About