Objective evaluation of deep uncertainty predictions for COVID-19 detection

Abstract

Deep neural networks (DNNs) have been widely applied for detecting COVID-19 in medical images. Existing studies mainly apply transfer learning and other data representation strategies to generate accurate point estimates. The generalization power of these networks is always questionable due to being developed using small datasets and failing to report their predictive confidence. Quantifying uncertainties associated with DNN predictions is a prerequisite for their trusted deployment in medical settings. Here we apply and evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray (CXR) images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced. Through comprehensive experiments, it is shown that networks pertained on CXR images outperform networks pretrained on natural image datasets such as ImageNet. Qualitatively and quantitatively evaluations also reveal that the predictive uncertainty estimates are statistically higher for erroneous predictions than correct predictions. Accordingly, uncertainty quantification methods are capable of flagging risky predictions with high uncertainty estimates. We also observe that ensemble methods more reliably capture uncertainties during the inference. DNN-based solutions for COVID-19 detection have been mainly proposed without any principled mechanism for risk mitigation. Previous studies have mainly focused on on generating single-valued predictions using pretrained DNNs. In this paper, we comprehensively apply and comparatively evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced for the first time. Using these new uncertainty performance metrics, we quantitatively demonstrate when we could trust DNN predictions for COVID-19 detection from chest X-rays. It is important to note the proposed novel uncertainty evaluation metrics are generic and could be applied for evaluation of probabilistic forecasts in all classification problems.

Prediction and forecasting of worldwide corona virus (COVID‐19) outbreak using time series and machine learning

“…The impact of the government's measures on the infection's rate of spread is indicated by the five important transition points in the growth curve of verified Indian cases. The chest X‐ray images are used to detect COVID‐19 by three uncertainty quantification strategies that are comparatively evaluated and comprehensively applied by Asgharnezhad et al 18 For the first time, new performance criteria for the objective evaluation of uncertainty estimations as well as a novel concept of uncertainty confusion matrix are provided. They quantitatively demonstrate when they could trust DNN predictions for COVID‐19 detection from chest X‐rays using these new uncertainty performance criteria.…”

LiteCovidNet: A lightweight deep neural network model for detection of COVID‐19 using X‐ray images

“… 2 This virus lately called Coronavirus Disease (COVID‐19), recognized world‐widely as Global Communal Health Extremity by World Health Organization (WHO) in January 2020, also acquired the status of “Pandemic” on March 11, 2020. 3 , 4 , 5 , 6 , 7 , 8 , 9 Till August 13, 2021, there are approximately 206 million confirmed cases across the world and 4.3 million deaths reported from more than 200 different countries and territories. The worst‐hit countries were the USA, India, and Brazil.…”

Automated Multi-View Multi-Modal Assessment of COVID-19 Patients Using Reciprocal Attention and Biomedical Transform

“…Ayoobi et al ( 27 ) proposed to predict new cases and death rates of COVID-19 patients in different time spans utilizing multiple deep learning methods. Asgharnezhad et al ( 28 ) proposed to quantify the competency of DNNs for generating reliable uncertainty estimates for COVID-19 diagnosis by introducing novel performance metrics. Alizadehsani et al ( 29 ) proposed to cope with insufficient labeled COVID-19 data by introducing a semi-supervised classification method relying on Sobel edge detection and generative adversarial networks (GANs).…”