Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration

Tomani, Christian; Cremers, Daniel; Buettner, Florian

doi:10.1007/978-3-031-19778-9_32

Cited by 10 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Initially, most of the work focused on either binary classification models [39], [40], [41], [42], [43], or models that used one-vs-all strategy to handle multi-classification problems [44], [45]. In recent years, with the popularity of artificial neural network algorithms, many methods have directly processed neural networks to achieve multi-classification calibration and achieve good results [5], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55].…”

Section: B Confidence Calibrationmentioning

confidence: 99%

A survey on confidence calibration of deep learning under class imbalance data

Dong,

Jiang,

Pan

et al. 2024

Preprint

View full text Add to dashboard Cite

Confidence calibration in classification models, a technique to achieve accurate posterior probability estimation for classification results, is crucial for assessing the likelihood of correct decisions in real-world applications. Class imbalance data, which biases the learning of the model and subsequently skews the posterior probabilities of the model, makes confidence calibration more challenging. Especially for often more important minority classes with high uncertainty, confidence calibration is more complex and necessary. Unlike previous surveys that typically separately investigate confidence calibration or class imbalance, this paper comprehensively investigates confidence calibration methods for deep learning-based classification models under class imbalance. Firstly, the problem of confidence calibration under class imbalance data is outlined. Secondly, a novel exploratory analysis regarding the impact of class imbalance data on confidence calibration is carried out, which can explain some experimental findings in existing studies. Then, this paper conducts a comprehensive review of 57 state-of-the-art confidence calibration methods under class imbalance data, divides these methods into six groups according to method differences, and systematically compares seven properties to evaluate their superiority. Subsequently, some commonly used and emerging evaluation methods in this field are summarized. Finally, we discuss several promising research directions that may serve as a guideline for future studies.

show abstract

Section: B Confidence Calibrationmentioning

confidence: 99%

A survey on confidence calibration of deep learning under class imbalance data

Dong,

Jiang,

Pan

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Standard neural networks typically yield non-calibrated predictions, which can be transformed into calibrated confidence scores using post-hoc calibration methods [30]. Nevertheless, modern neural networks tend to yield systematically overconfident predictions [13].…”

Section: Expected Calibration Error (Ece)mentioning

confidence: 99%

Sensitivity analysis of AI-based algorithms for autonomous driving on optical wavefront aberrations induced by the windshield

Wolf,

Ulrich,

Kapoor

2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Autonomous driving perception techniques are typically based on supervised machine learning models that are trained on real-world street data. A typical training process involves capturing images with a single car model and windshield configuration. However, deploying these trained models on different car types can lead to a domain shift, which can potentially hurt the neural networks performance and violate working ADAS requirements. To address this issue, this paper investigates the domain shift problem further by evaluating the sensitivity of two perception models to different windshield configurations. This is done by evaluating the dependencies between neural network benchmark metrics and optical merit functions by applying a Fourier optics based threat model. Our results show that there is a performance gap introduced by windshields and existing optical metrics used for posing requirements might not be sufficient.

show abstract

“…However, it is regrettable that the current studies [9][10][11] on multimodal sentiment analysis rarely focuses on uncertainty calibration, thus overlooking the crucial significance of uncertainty calibration in improving model performance. Recently, there have been studies [12,13] focusing on improving model performance from the perspective of uncertainty calibration, whereas these methods often discuss the issue from an unimodal perspective, neglecting the challenge of inconsistent sentiment polarities across different modalities, which poses a new challenge to uncertainty calibration. Therefore, it is necessary to conduct further research to explore how to achieve effective uncertainty calibration in multimodal sentiment analysis.…”

Section: Introductionmentioning

confidence: 99%

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

Pan,

Meng

2024

Electronics

View full text Add to dashboard Cite

In open environments, multimodal sentiment analysis (MSA) often suffers from low-quality data and can be disrupted by noise, inherent defects, and outliers. In some cases, unreasonable multimodal fusion methods can perform worse than unimodal methods. Another challenge of MSA is effectively enabling the model to provide accurate prediction when it is confident and to indicate high uncertainty when its prediction is likely to be inaccurate. In this paper, we propose an uncertain-aware late fusion based on hybrid uncertainty calibration (ULF-HUC). Firstly, we conduct in-depth research on the issue of sentiment polarity distribution in MSA datasets, establishing a foundation for an uncertain-aware late fusion method, which facilitates organic fusion of modalities. Then, we propose a hybrid uncertainty calibration method based on evidential deep learning (EDL) that balances accuracy and uncertainty, supporting the reduction of uncertainty in each modality of the model. Finally, we add two common types of noise to validate the effectiveness of our proposed method. We evaluate our model on three publicly available MSA datasets (MVSA-Single, MVSA-Multiple, and MVSA-Single-Small). Our method outperforms state-of-the-art approaches in terms of accuracy, weighted F1 score, and expected uncertainty calibration error (UCE) metrics, proving the effectiveness of the proposed method.

show abstract

Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration

Cited by 10 publications

References 9 publications

A survey on confidence calibration of deep learning under class imbalance data

A survey on confidence calibration of deep learning under class imbalance data

Sensitivity analysis of AI-based algorithms for autonomous driving on optical wavefront aberrations induced by the windshield

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

Contact Info

Product

Resources

About