An Uncertainty-aware Loss Function for Training Neural Networks with Calibrated Predictions

Shamsi, Afshar; Asgharnezhad, Hamzeh; Tajally, AmirReza; Nahavandi, Saeid; Leung, Henry

doi:10.48550/arxiv.2110.03260

Cited by 2 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Authors in [67] have proposed an interesting idea for optimizing a simple MCD model, which yielded higher Uacc for the model without sacrificing model accuracy. To this end, a specific loss function is used that takes into account both Uacc and the accuracy of the model:…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Quantifying Uncertainty in Automated Detection of Alzheimer’s Patients Using Deep Neural Network

Roshanzamir¹,

Shamsi²,

Asgharnezhad³

et al. 2023

Preprint

View full text Add to dashboard Cite

One of the most common forms of dementia is Alzheimer’s disease (AD), which leads to progressive mental deterioration. Unfortunately, there is no definitive diagnosis and cure that can stop the condition progressing. The diagnosis is often performed based on the clinical history and neuropsychological data, including magnetic resonance imaging (MRI). Deep neural networks (DNN) algorithms are gaining popularity for medical diagnosis, and have been used widely for the analysis of MRI data. DNNs can extract hidden features from thousands of training images automatically. However, they cannot judge how confident they are about their predictions. To use DNNs in safety-critical applications such as medical diagnosis, uncertainty quantification of DNNs predictions is crucial. For this purpose, Monte Carlo dropout (MCD) has been widely used, however, it may lead to overconfident and miss calibrated results. This paper proposes a framework in which the MCD algorithm’s hyper-parameters are optimized during training using Bayesian optimization for the first time. The conducted optimization leads to assigning high predictive entropy to erroneous predictions and making it possible to recognize risky predictions. The proposed framework is used for AD diagnosis, which has not been done before. We compare our method with some existing methods in the literature based on different uncertainty quantification criteria. The results of comprehensive experiments on the Kaggle dataset using a deep model pre-trained on the ImageNet dataset show that the proposed algorithm can quantify uncertainty much better than the existing methods.

show abstract

Section: Resultsmentioning

confidence: 99%

“…In an ideal scenario, high uncertainty is assigned to model predictions whenever the model is not confident about its predictions. In [67], the uncertainty-aware C loss function was introduced to improve MCD performance. Although the results were acceptable, they tunned hyperparameters by hand.…”

Section: Datasetmentioning

confidence: 99%

Quantifying Uncertainty in Automated Detection of Alzheimer’s Patients Using Deep Neural Network

Roshanzamir¹,

Shamsi²,

Asgharnezhad³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In [26], the targeted dropout was suggested to leverage robustness to subsequent pruning when training a large and sparse network using a simple self-reinforcing sparsity criterion. In [27], authors combined Expected Calibration Error (ECE) and Predictive Entropy (PE) with cross-entropy to form two new loss functions. The proposed loss functions improve the uncertainty estimate of the MC dropout model.…”

Section: Introductionmentioning

confidence: 99%

Controlled Dropout for Uncertainty Estimation

Hasan¹,

Khosravi²,

Hossain³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Uncertainty quantification in a neural network is one of the most discussed topics for safety-critical applications. Though Neural Networks (NNs) have achieved state-of-the-art performance for many applications, they still provide unreliable point predictions, which lack information about uncertainty estimates. Among various methods to enable neural networks to estimate uncertainty, Monte Carlo (MC) dropout has gained much popularity in a short period due to its simplicity. In this study, we present a new version of the traditional dropout layer where we are able to fix the number of dropout configurations. As such, each layer can take and apply the new dropout layer in the MC method to quantify the uncertainty associated with NN predictions. We conduct experiments on both toy and realistic datasets and compare the results with the MC method using the traditional dropout layer. Performance analysis utilizing uncertainty evaluation metrics corroborates that our dropout layer offers better performance in most cases.

show abstract

An Uncertainty-aware Loss Function for Training Neural Networks with Calibrated Predictions

Cited by 2 publications

References 29 publications

Quantifying Uncertainty in Automated Detection of Alzheimer’s Patients Using Deep Neural Network

Quantifying Uncertainty in Automated Detection of Alzheimer’s Patients Using Deep Neural Network

Controlled Dropout for Uncertainty Estimation

Contact Info

Product

Resources

About