Delving Deep into Label Smoothing

Zhang, Chang-Bin; Jiang, Peng-Tao; Hou, Qibin; Wei, Yunchao; Han, Qi; Li, Zhen; Cheng, Ming-Ming

doi:10.48550/arxiv.2011.12562

Cited by 3 publications

(3 citation statements)

References 51 publications

(90 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VII, we report the accuracy for three common computer vision datasets, including CIFAR10, CIFAR100, and Tiny-ImageNet. Compared to CE, LS [19], Online LS [53] and Disturb Label [62], our CBLS shows superior results.…”

Section: Ablation Studymentioning

confidence: 97%

Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding

Islam

Glocker

et al. 2024

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

Curriculum learning and self-paced learning are the training strategies that gradually feed the samples from easy to more complex. They have captivated increasing attention due to their excellent performance in robotic vision. Most recent works focus on designing curricula based on difficulty levels in input samples or smoothing the feature maps. However, smoothing labels to control the learning utility in a curriculum manner is still unexplored. In this work, we design a paced curriculum by label smoothing (P-CBLS) using paced learning with uniform label smoothing (ULS) for classification tasks and fuse uniform and spatially varying label smoothing (SVLS) for semantic segmentation tasks in a curriculum manner. In ULS and SVLS, a bigger smoothing factor value enforces a heavy smoothing penalty in the true label and limits learning less information. Therefore, we design the curriculum by label smoothing (CBLS). We set a bigger smoothing value at the beginning of training and gradually decreased it to zero to control the model learning utility from lower to higher. We also designed a confidenceaware pacing function and combined it with our CBLS to investigate the benefits of various curricula. The proposed techniques are validated on four robotic surgery datasets of multiclass, multi-label classification, captioning, and segmentation tasks. We also investigate the robustness of our method by corrupting validation data into different severity levels. Our extensive analysis shows that the proposed method improves prediction accuracy and robustness. The code is publicly available at https://github.com/XuMengyaAmy/P-CBLS.

show abstract

Section: Ablation Studymentioning

confidence: 97%

Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding

Islam

Glocker

et al. 2024

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

show abstract

“…The approach has been tested on a dataset made of sound recordings and a remarkable performance has been achieved [16]. Zhang et al have proposed a label smoothing approach to generate soft labels by applying weighted average between distribution and hard labels [17].…”

Section: Related Workmentioning

confidence: 99%

Game Theory-Based Dynamic Weighted Ensemble for Retinal Disease Classification

Mittal¹,

Rajam²

2023

Intelligent Automation &Amp; Soft Computing

View full text Add to dashboard Cite

An automated retinal disease detection system has long been in existence and it provides a safe, no-contact and cost-effective solution for detecting this disease. This paper presents a game theory-based dynamic weighted ensemble of a feature extraction-based machine learning model and a deep transfer learning model for automatic retinal disease detection. The feature extractionbased machine learning model uses Gaussian kernel-based fuzzy rough sets for reduction of features, and XGBoost classifier for the classification. The transfer learning model uses VGG16 or ResNet50 or Inception-ResNet-v2. A novel ensemble classifier based on the game theory approach is proposed for the fusion of the outputs of the transfer learning model and the XGBoost classifier model. The ensemble approach significantly improves the accuracy of retinal disease prediction and results in an excellent performance when compared to the individual deep learning and feature-based models.

show abstract

“…Other works that use label smoothing as a calibration benchmark [10,12,13] very briefly discuss validating the smoothing factor, however it is generally minimal with at most three values considered and the chosen smoothing factor is then used across all experiments (model architectures, datasets, training setups). Zhang et al [15] propose an online label smoothing strategy for image classification that implicitly measures class similarity and generates soft labels based on the statistics of the model prediction for the target category. In contrast, Liu and JaJa [25], calculates class similarity explicitly prior to training and uses these scores to obtain smoothing factors for each target class.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Confidence Calibration

Pearce¹,

Meger²

2022

Proceedings of the Canadian Conference on Artificial Intelligence

View full text Add to dashboard Cite

Confidence calibration is a quickly growing area of research in deep learning, including computer vision applications. New model architectures and loss functions are being introduced to improve model calibration, an important topic for safety critical AI applications. In order to evaluate these new methods, they are frequently compared to simple baselines, which serve as a comparison and help measure new methods' effectiveness. Popular baselines for evaluating model confidence calibration include label smoothing, mixup and dropout. Despite these methods frequent use and simple implementations, the parameter values that define them are rarely validated, and are usually a default value. This paper demonstrates the danger in using these default values for calibration benchmarks on common datasets; poor model calibration, specifically, a model that is purely over-or underconfident. We present an adaptive framework that can adjust the parameter value of these baseline methods during training based on validation accuracy and confidence to maintain good model calibration and balance over-and underconfidence. A disadvantage of binning-based calibration metrics is that they are not differentiable and therefore cannot be added to the training loss to penalize calibration error. Our adaptive framework solves this problem and provides a solution for optimizing calibration methods. Experiments with the CIFAR-10 and CIFAR-100 image classification datasets show that our approach improves model calibration compared to using these popular methods with default values and we achieve good model calibration regardless of model architecture and dataset. Further, our analysis provides a comparison between these baseline methods that can be considered for future confidence calibration research.

show abstract

Delving Deep into Label Smoothing

Cited by 3 publications

References 51 publications

Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding

Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding

Game Theory-Based Dynamic Weighted Ensemble for Retinal Disease Classification

Adaptive Confidence Calibration

Contact Info

Product

Resources

About