Objective Evaluation of Deep Uncertainty Predictions for COVID-19 Detection

Abstract

Deep neural networks (DNNs) have been widely applied for detecting COVID-19 in medical images. Existing studies mainly apply transfer learning and other data representation strategies to generate accurate point estimates. The generalization power of these networks is always questionable due to being developed using small datasets and failing to report their predictive confidence. Quantifying uncertainties associated with DNN predictions is a prerequisite for their trusted deployment in medical settings. Here we apply and evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray (CXR) images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced. Through comprehensive experiments, it is shown that networks pertained on CXR images outperform networks pretrained on natural image datasets such as ImageNet. Qualitatively and quantitatively evaluations also reveal that the predictive uncertainty estimates are statistically higher for erroneous predictions than correct predictions. Accordingly, uncertainty quantification methods are capable of flagging risky predictions with high uncertainty estimates. We also observe that ensemble methods more reliably capture uncertainties during the inference. DNN-based solutions for COVID-19 detection have been mainly proposed without any principled mechanism for risk mitigation. Previous studies have mainly focused on on generating single-valued predictions using pretrained DNNs. In this paper, we comprehensively apply and comparatively evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced for the first time. Using these new uncertainty performance metrics, we quantitatively demonstrate where and when we could trust DNN predictions for COVID-19 detection from chest X-rays. It is important to note the proposed novel uncertainty evaluation metrics are generic and could be applied for evaluation of probabilistic forecasts in all classification problems.

Overhead Reduction Technique for Software-Defined Network Based Intrusion Detection Systems

“…Bayes' Theorem utilises to compute the posterior probability for the given classes. Equation 1 has been used to calculate the posterior probability [25]:…”

“…Where P(c|x) indicates the probability of the predictor group c (x, features), P(c) represents the preceding probability of the class; P(x|c) represents the probability of a feature in the current class. P(x) defines the likelihood of a feature preceding it [25].…”

Controlled Dropout for Uncertainty Estimation

“…Normally, model predictions are grouped into two categories: correct and incorrect. For uncertainty estimation, we have another two groups: certain and uncertain predictions [33]. We also need to have some threshold values to categorize predictions into certain and uncertain.…”

“…Confusion matrix is an effective tool to estimate the performance of a model in classification problems where accuracy alone can be a misleading indicator due to the uneven distributions of classes. Performance metrics similar to regular confusion matrix metrics are also defined for uncertainty estimation as below [33]:…”

Requirement analysis for an artificial intelligence model for the diagnosis of the COVID-19 from chest X-ray data

“…Reference [55] calls for uncertainty evaluation for predictions. This is related to lack of statistical analysis of the prediction quality mentioned in both references [3] and [4].…”