Uncertainty Quantification in Computer-Aided Diagnosis: Make Your Model say "I don't know" for Ambiguous Cases

Laves, Max-Heinrich; Ihler, Sontje; Ortmaier, Tobias

doi:10.48550/arxiv.1908.00792

Cited by 4 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3). In fact, uncertainty calculated using the Bayesian approach tends to be uncalibrated [48], [51], [52], and further processes are required to achieve calibrated uncertainty measures. In this work, we chose to use Uncertainty Calibration Error (UCE), developed by [48].…”

Section: Performance Results and Analysismentioning

confidence: 99%

Forecasting Lightpath Quality of Transmission and Implementing Uncertainty in the Forecast Models

Yousefi

Chouman

Djukic

et al. 2023

J. Lightwave Technol.

View full text Add to dashboard Cite

The recent popularity of using deep learning models for the forecasting of time series calls for methods to not only predict the target but also measure the uncertainty of the prediction accurately. Working with time series requires reliable and stable forecasters. An essential component of the reliability of machine learning (ML) and deep learning (DL) models is the estimation of the uncertainty. In this work, we address building and characterizing time series forecasters, including N-Beats, Long Short-Term Memory (LSTM) and Multilayer Perceptron (MLP) against the Naive model, and define the confidence margins, and uncertainty for the selected model.All the implementations are conducted in Python programming language. Random sampling is performed to avoid overfitting. Our target field data is North American Service Provider data sets (NASP). Among the implemented models, the MLP model is selected to measure the uncertainty and confidence level, and the Monte Carlo dropout, which approximates Bayesian uncertainty, is applied during inference to render the implementation of uncertainty calculations. Quantile Regression is also implemented on the MLP algorithm as a baseline to predict the confidence intervals and to evaluate our strategy for estimating uncertainty. To establish reliable uncertainty estimation in time series predictions, we performed uncertainty calibration. Motivated by recent developments in Expected Uncertainty Calibration Error (UCE), we modified the uncertainty calculated by the probabilistic Bayesian estimations. Detailed experiments and architectures of the solution are presented.

show abstract

Section: Performance Results and Analysismentioning

confidence: 99%

Forecasting Lightpath Quality of Transmission and Implementing Uncertainty in the Forecast Models

Yousefi

Chouman

Djukic

et al. 2023

J. Lightwave Technol.

View full text Add to dashboard Cite

show abstract

“…Assuming the model's prediction is always correct without any reasoning on the model's uncertainty may result in catastrophic results. This fact led the researchers to suggest abstaining models based on certain conditions like when the model's uncertainty is high, thus improving the reliability [41,42].…”

Section: Preliminariesmentioning

confidence: 99%

Uncertainty as a Swiss army knife: new adversarial attack and defense ideas based on epistemic uncertainty

Tuna

Çatak

Eskil

2022

Complex Intell. Syst.

View full text Add to dashboard Cite

Although state-of-the-art deep neural network models are known to be robust to random perturbations, it was verified that these architectures are indeed quite vulnerable to deliberately crafted perturbations, albeit being quasi-imperceptible. These vulnerabilities make it challenging to deploy deep neural network models in the areas where security is a critical concern. In recent years, many research studies have been conducted to develop new attack methods and come up with new defense techniques that enable more robust and reliable models. In this study, we use the quantified epistemic uncertainty obtained from the model’s final probability outputs, along with the model’s own loss function, to generate more effective adversarial samples. And we propose a novel defense approach against attacks like Deepfool which result in adversarial samples located near the model’s decision boundary. We have verified the effectiveness of our attack method on MNIST (Digit), MNIST (Fashion) and CIFAR-10 datasets. In our experiments, we showed that our proposed uncertainty-based reversal method achieved a worst case success rate of around 95% without compromising clean accuracy.

show abstract

“…One of the major reasons behind is the lack of reliability of existing CAD systems. Even though CAD has been studied widely, the uncertainty estimation of DNNs in medical imaging is remarkably understudied (Laves, Ihler, and Ortmaier 2019;Poduval, Loya, and Sethi 2020). Hence, in this paper we aim to conduct a comprehensive study on mitigating uncertainty of DNNs on COVID-19 detection.…”

Section: Introductionmentioning

confidence: 99%

Reducing Risk and Uncertainty of Deep Neural Networks on Diagnosing COVID-19 Infection

Sarker¹,

Pandit²,

Sarker³

et al. 2021

Preprint

View full text Add to dashboard Cite

Effective and reliable screening of patients via Computer-Aided Diagnosis can play a crucial part in the battle against COVID-19. Most of the existing works focus on developing sophisticated methods yielding high detection performance, yet not addressing the issue of predictive uncertainty. In this work, we introduce uncertainty estimation to detect confusing cases for expert referral to address the unreliability of stateof-the-art (SOTA) DNNs on COVID-19 detection. To the best of our knowledge, we are the first to address this issue on the COVID-19 detection problem. In this work, we investigate a number of SOTA uncertainty estimation methods on publicly available COVID dataset and present our experimental findings. In collaboration with medical professionals, we further validate the results to ensure the viability of the best performing method in clinical practice.

show abstract

Uncertainty Quantification in Computer-Aided Diagnosis: Make Your Model say "I don't know" for Ambiguous Cases

Cited by 4 publications

References 4 publications

Forecasting Lightpath Quality of Transmission and Implementing Uncertainty in the Forecast Models

Forecasting Lightpath Quality of Transmission and Implementing Uncertainty in the Forecast Models

Uncertainty as a Swiss army knife: new adversarial attack and defense ideas based on epistemic uncertainty

Reducing Risk and Uncertainty of Deep Neural Networks on Diagnosing COVID-19 Infection

Contact Info

Product

Resources

About