On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking

Aydin, Orhun Utku; Taha, Abdel Aziz; Hilbert, Adam; Khalil, Ahmed A.; Galinović, Ivana; Fiebach, Jochen B.; Frey, Dietmar; Madai, Vince I.

doi:10.1186/s41747-020-00200-2

Cited by 90 publications

(52 citation statements)

References 13 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On one hand, research has focused on developing completely new [ 30 ], modified [ 26 ] or combined [ 14 ] performance measures that are more sensitive to errors and have wider score ranges to distinguish between subtle differences between ground truth and segmentation. For example, Chang et al proposed Conformity instead of DICE and Sensibility instead of Specificity.…”

Section: Discussionmentioning

confidence: 99%

“…EvaluateSegmentation is an evaluation framework for medical image segmentation comprising implementation of various performance measures from the literature to assess segmentation quality. In addition to the average Hausdorff distance, the tool also included an improved version of the average Hausdorff distance called the balanced average Hausdorff Distance that was introduced recently [ 26 ]. The 95th quantile of the Hausdorff distance was utilised to handle outliers [ 27 ].…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

An evaluation of performance measures for arterial brain vessel segmentation

et al. 2021

Self Cite

View full text Add to dashboard Cite

Background Arterial brain vessel segmentation allows utilising clinically relevant information contained within the cerebral vascular tree. Currently, however, no standardised performance measure is available to evaluate the quality of cerebral vessel segmentations. Thus, we developed a performance measure selection framework based on manual visual scoring of simulated segmentation variations to find the most suitable measure for cerebral vessel segmentation. Methods To simulate segmentation variations, we manually created non-overlapping segmentation errors common in magnetic resonance angiography cerebral vessel segmentation. In 10 patients, we generated a set of approximately 300 simulated segmentation variations for each ground truth image. Each segmentation was visually scored based on a predefined scoring system and segmentations were ranked based on 22 performance measures common in the literature. The correlation of visual scores with performance measure rankings was calculated using the Spearman correlation coefficient. Results The distance-based performance measures balanced average Hausdorff distance (rank = 1) and average Hausdorff distance (rank = 2) provided the segmentation rankings with the highest average correlation with manual rankings. They were followed by overlap-based measures such as Dice coefficient (rank = 7), a standard performance measure in medical image segmentation. Conclusions Average Hausdorff distance-based measures should be used as a standard performance measure in evaluating cerebral vessel segmentation quality. They can identify more relevant segmentation errors, especially in high-quality segmentations. Our findings have the potential to accelerate the validation and development of novel vessel segmentation approaches.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

An evaluation of performance measures for arterial brain vessel segmentation

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the 20 node-positive scans, the 5 testing segmentation masks for each HN-CT scan were combined to create a “consensus segmentation mask” using the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm ( Figure 3 ) 16 . The testing segmentation masks and consensus segmentation masks were compared to their respective ground-truth masks using overlap-based (DSC), volume-based (volume similarity), spatial distance-based (Hausdorff distance [HD]), and probabilistic-based (Cohen Kappa Coefficient [CKC]) metrics 17 .…”

Section: Methodsmentioning

confidence: 99%

Auto-Detection and Segmentation of Involved Lymph Nodes in HPV-Associated Oropharyngeal Cancer Using a Convolutional Deep Learning Neural Network

Taku¹,

Wahid²,

Dijk³

et al. 2022

Preprint

View full text Add to dashboard Cite

Purpose: Segmentation of involved lymph nodes on head and neck computed tomography (HN-CT) scans is necessary for the radiotherapy treatment planning of human papilloma virus (HPV) associated oropharynx cancers (OPC). We aimed to train a deep learning convolutional neural network (DL-CNN) to identify and segment involved lymph nodes on contrast-enhanced HN-CT scans. Methods: 90 patients who underwent levels II-IV neck dissection for newly diagnosed, clinically node-positive, HPV-OPC were identified. Ground-truth segmentation of all radiographically and pathologically involved nodes was manually performed on pre-surgical HN-CT scans, which were randomly divided into training/validation dataset (n=70) and testing dataset (n=20). A 5-fold cross validation was used to train 5 separate DL-CNN sub-models based on a residual U-net architecture. Validation and testing segmentation masks were compared to ground-truth segmentation masks using overlap-based, volume-based, and distance-based metrics. A lymph auto-detection model was developed by thresholding segmentation model outputs, and 20 node-negative HN-CT scans were added to the test set to further evaluate auto-detection capabilities. Model discrimination of lymph node positive and negative HN-CT scans was evaluated using the area under the receiver operating characteristic curve (AUC). Results: In the DL-CNN validation phase, all sub-models yielded segmentation masks with median DSC > 0.90 and median volume similarity score of > 0.95. In the testing phase, the DL-CNN produced consensus segmentation masks with median Dice of 0.92 (IQR, 0.89-0.95), median volume similarity of 0.97 (IQR, 0.94-0.99), and median Hausdorff distance of 4.52 mm (IQR, 1.22-8.38). The detection model achieved an AUC of 0.98. Conclusion: The results from this single-institution study demonstrate the successful automation of lymph node segmentation for patients with HPV-OPC using a DL-CNN. Future studies, including external validation using a larger dataset, are necessary to clarify the role of the DL-CNN in the routine radiation oncology treatment planning workflow.

show abstract

“…As the DSC quantifies the overlap of the ground truth and prediction scaled by the total number of voxels in ground truth and prediction, it is a robust performance measure for imbalanced segmentations, i.e., images contain more background than segmented area. The bAHD is a newly proposed metric for evaluating segmentations (Aydin et al, 2021 ):…”

Section: Methodsmentioning

confidence: 99%

Toward Sharing Brain Images: Differentially Private TOF-MRA Images With Segmentation Labels Using Generative Adversarial Networks

Kossen¹,

Hirzel²,

Madai³

et al. 2022

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Sharing labeled data is crucial to acquire large datasets for various Deep Learning applications. In medical imaging, this is often not feasible due to privacy regulations. Whereas anonymization would be a solution, standard techniques have been shown to be partially reversible. Here, synthetic data using a Generative Adversarial Network (GAN) with differential privacy guarantees could be a solution to ensure the patient's privacy while maintaining the predictive properties of the data. In this study, we implemented a Wasserstein GAN (WGAN) with and without differential privacy guarantees to generate privacy-preserving labeled Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) image patches for brain vessel segmentation. The synthesized image-label pairs were used to train a U-net which was evaluated in terms of the segmentation performance on real patient images from two different datasets. Additionally, the Fréchet Inception Distance (FID) was calculated between the generated images and the real images to assess their similarity. During the evaluation using the U-Net and the FID, we explored the effect of different levels of privacy which was represented by the parameter ϵ. With stricter privacy guarantees, the segmentation performance and the similarity to the real patient images in terms of FID decreased. Our best segmentation model, trained on synthetic and private data, achieved a Dice Similarity Coefficient (DSC) of 0.75 for ϵ = 7.4 compared to 0.84 for ϵ = ∞ in a brain vessel segmentation paradigm (DSC of 0.69 and 0.88 on the second test set, respectively). We identified a threshold of ϵ <5 for which the performance (DSC <0.61) became unstable and not usable. Our synthesized labeled TOF-MRA images with strict privacy guarantees retained predictive properties necessary for segmenting the brain vessels. Although further research is warranted regarding generalizability to other imaging modalities and performance improvement, our results mark an encouraging first step for privacy-preserving data sharing in medical imaging.

show abstract

On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking

Cited by 90 publications

References 13 publications

An evaluation of performance measures for arterial brain vessel segmentation

An evaluation of performance measures for arterial brain vessel segmentation

Auto-Detection and Segmentation of Involved Lymph Nodes in HPV-Associated Oropharyngeal Cancer Using a Convolutional Deep Learning Neural Network

Toward Sharing Brain Images: Differentially Private TOF-MRA Images With Segmentation Labels Using Generative Adversarial Networks

Contact Info

Product

Resources

About