Quantifying the uncertainty of deep learning-based computer-aided diagnosis for patient safety

Abstract

In this work, we discuss epistemic uncertainty estimation obtained by Bayesian inference in diagnostic classifiers and show that the prediction uncertainty highly correlates with goodness of prediction. We train the ResNet-18 image classifier on a dataset of 84,484 optical coherence tomography scans showing four different retinal conditions. Dropout is added before every building block of ResNet, creating an approximation to a Bayesian classifier. Monte Carlo sampling is applied with dropout at test time for uncertainty estimation. In Monte Carlo experiments, multiple forward passes are performed to get a distribution of the class labels. The variance and the entropy of the distribution is used as metrics for uncertainty. Our results show strong correlation with ρ = 0.99 between prediction uncertainty and prediction error. Mean uncertainty of incorrectly diagnosed cases was significantly higher than mean uncertainty of correctly diagnosed cases. Modeling of the prediction uncertainty in computer-aided diagnosis with deep learning yields more reliable results and is therefore expected to increase patient safety. This will help to transfer such systems into clinical routine and to increase the acceptance of machine learning in diagnosis from the standpoint of physicians and patients.

Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection

“…In order to assess the quality of predictive uncertainty, we leveraged Spearman's correlation coefficient between Predictive Entropy (PH) and Bayesian Active Learning by Disagreement (BALD). We quantified the predictive accuracy by 1-Wasserstein distance (WD) to measure how much the estimated uncertainty correlates with the true errors [2,14]. The Wasserstein distance for the real data distribution P r and the generated data distribution P g is mathematically defined as the greatest lower bound (infimum) for any transport plan (i.e.…”

“…It shows that the estimated uncertainty is higher for erroneous predictions. Therefore, uncertainty information provides as an additional insight to point prediction to refer the uncertain images to radiologists for further investigation [14], which improves the overall prediction performance.…”

“…Fig.5. Correlation between estimated predictive entropy as a measure of Uncertainty and Accuracy in prediction[14] …”

Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

“…Uncertainty estimation [17] is emerging as a very important topic in machine learning with a wide range of applications [4,11,27,32,51,54]. For example, in [54], uncertainty sampling was employed to increase the diversity of selected training data.…”

“…In [16], DPV was used for active learning applications, as active learning methods generally rely on uncertainty scores which guides learning and updates of models from small amounts of training data. [32] used DPV to improve computer-aided diagnoses and their robustness for patient safety.…”