Uncertainty estimation for deep learning-based pectoral muscle segmentation via Monte Carlo dropout

Abstract

Objective: Deep Learning models are often susceptible to failures after deployment. Knowing when your model is producing inadequate predictions is crucial. In this work, we investigate the utility of Monte Carlo (MC) dropout and the efficacy of the proposed uncertainty metric (UM) for flagging of unacceptable pectoral muscle segmentations in mammograms. 

Approach: Segmentation of pectoral muscle was performed with modified ResNet18 Convolutional Neural Network (CNN). MC dropout layers were kept unlocked at inference time. For each mammogram, 50 pectoral muscle segmentations were generated. The mean was used to produce the final segmentation and the standard deviation was applied for the estimation of uncertainty. From each pectoral muscle uncertainty map, the overall UM was calculated. To validate the UM, a correlation between the dice similarity coefficient (DSC) and UM was used. The UM was first validated in a training set (200 mammograms) and finally tested in an independent dataset (300 mammograms). ROC-AUC analysis was performed to test the discriminatory power of the proposed UM for flagging unacceptable segmentations.

Main results: The introduction of dropout layers in the model improved segmentation performance (DSC=0.95±0.07 vs. DSC=0.93±0.10). Strong anti-correlation (r=-0.76, p<0.001) between the proposed UM and DSC was observed. A high AUC of 0.98 (97% specificity at 100% sensitivity) was obtained for the discrimination of unacceptable segmentations. Qualitative inspection by the radiologist revealed that images with high UM are difficult to segment. 

Significance: The use of Monte Carlo dropout at inference time in combination with the proposed uncertainty metric enables flagging of unacceptable pectoral muscle segmentations from mammograms with excellent discriminatory power.

Uncertainty quantification via localized gradients for deep learning-based medical image assessments

“…Assessments involving the association between deep learning-based uncertainty and reader-based uncertainty are not common, likely due to the burdensome challenge of acquiring reader-based uncertainty. In the work of (Klanecek et al 2023), an uncertainty measure was similarly related to qualitative reader assessments, however, these assessments consisted of the 'acceptability' of deep learning-based delineations. Both of these approaches are subject to reader errors and biases.…”

“…In (G. Wang et al 2019), test-time data augmentation was used to generate uncertainty measures that corresponded with delineation performance in MRI data. MC dropout-based methods similarly have been used to flag poorly segmented test images (McClure et al 2019, Mehrtash et al 2020, Nair et al 2020, Klanecek et al 2023, Ng et al 2023. Following the logic behind the Deep Ensembles, (Kushibar et al 2022) constructed ensembled delineation outputs from various layers within a single network.…”