Objective Evaluation of Deep Uncertainty Predictions for COVID-19 Detection

Abstract

Deep neural networks (DNNs) have been widely applied for detecting COVID-19 in medical images. Existing studies mainly apply transfer learning and other data representation strategies to generate accurate point estimates. The generalization power of these networks is always questionable due to being developed using small datasets and failing to report their predictive confidence. Quantifying uncertainties associated with DNN predictions is a prerequisite for their trusted deployment in medical settings. Here we apply and evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray (CXR) images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced. Through comprehensive experiments, it is shown that networks pertained on CXR images outperform networks pretrained on natural image datasets such as ImageNet. Qualitatively and quantitatively evaluations also reveal that the predictive uncertainty estimates are statistically higher for erroneous predictions than correct predictions. Accordingly, uncertainty quantification methods are capable of flagging risky predictions with high uncertainty estimates. We also observe that ensemble methods more reliably capture uncertainties during the inference.Impact Statement-DNN-based solutions for COVID-19 detection have been mainly proposed without any principled mechanism for risk mitigation. The focus of existing literature is mainly on generating single-valued predictions using pretrained DNNs. In this paper, we comprehensively apply and comparatively evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced for the first time. Using these new uncertainty performance metrics, we quantitatively demonstrate where and when we could trust DNN predictions for COVID-19 detection from chest X-rays. It is important to note the proposed novel uncertainty evaluation metrics are generic and could be applied for evaluation of probabilistic forecasts in all classification problems.

An Uncertainty-aware Loss Function for Training Neural Networks with Calibrated Predictions

“…Wang and Rocková [11] 2020 N/A N/A Semi-parametric BvM (Bernstein-von Mises theorem) Chen et al [12] 2020 Text analysis Classification N/A Stoean et al [13] 2020 Medical Classification MC dropout Hirschfeld et al [14] 2020 Molecular property Prediction N/A Huo et al [15] 2020 Mobile activity Recognition MEL (maximum entropy learning) Schwaiger et al [16] 2020 Vision and image processing Out-of-distribution (OOD) EDL (Evidential Deep Learning) LC (Learned Confidence) Edupuganti et al [17] 2020 Medical Segmentation Monte-Carlo sampling Aseeri [18] 2021 Medical Classification MC dropout Shamsi et al [19] 2021 Medical Classification Bayesian Ensemble Hoffmann et al [20] 2021 Computational optical Segmentation Ensemble learning Abdar et al [21] 2021 Medical Classification TWDBDL (Three-Way Decision-based Bayesian DL) Phan et al [22] 2021 Sleep staging Classification Entropy-based confidence quantification Fig. (2) The uncertainty confusion matrix [35].…”

“…1. Based on this, Authors in [35] introduced a set of metrics to quantitatively and objectively evaluate uncertainty estimates generated by machine learning models. The most important uncertainty performance metrics are the uncertainty sensitivity (USen), the uncertainty specificity (USpe), the uncertainty precision, and the uncertainty accuracy (UA).…”

“…The most important uncertainty performance metrics are the uncertainty sensitivity (USen), the uncertainty specificity (USpe), the uncertainty precision, and the uncertainty accuracy (UA). All these four metrics are calculated based on the uncertainty confusion matrix [35]. The greater these metrics, the better the model performance in capturing and communicating aleatoric and epistemic uncertainties.…”

Uncertainty-Aware Credit Card Fraud Detection Using Deep Learning

“…The weakness of the classifier is conspicuous through FP and FN. Additionally, to evaluate the predictive uncertainty estimates an idea such as the confusion matrix was proposed [49]. A threshold is used to cast predictions into certain and uncertain categories.…”

Confidence Aware Neural Networks for Skin Cancer Detection

“…It should be noted that EMCD is a quite new method that uses the advantages of both the MCD technique and ensemble network. We use novel performance metrics for the quantitative and comprehensive evaluation of uncertainty predictions [19]. The uncertainty prediction evaluation is estimated in a similar way to that of binary classification evaluation.…”