Automated quality control in image segmentation: application to the UK Biobank cardiovascular magnetic resonance imaging study

Abstract

Background The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99% accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95% classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.

Improving the Generalizability of Convolutional Neural Network-Based Segmentation on CMR Images

“…Last but not least, for clinical deployment, it is necessary to alert users when failure happens. In this regard, future work can be integrating the segmentation approach with an automatic quality control module, providing automatic segmentation assessment [e.g., estimated segmentation scores (39), model uncertainty maps (40)] to clinicians for further verification and refinement.…”

Deep Learning for Cardiac Image Segmentation: A Review

“…Hybrid segmentation methods: Another stream of work aims at combining neural networks with classical segmentation approaches, e.g. levelsets (Ngo et al, 2017;Duan et al, 2018a), deformable models (Avendi et al, 2016(Avendi et al, , 2017Medley et al, 2019), atlas-based methods (Yang et al, 2016;Rohé et al, 2017) and graph-cut based methods . Here, neural networks are applied in the feature extraction and model initialization stages, reducing the dependency on manual interactions and improving the segmentation accuracy of the conventional segmentation methods deployed afterwards.…”

Artificial Intelligence in Nuclear Medicine

“…Thus, a human operator often cannot decide whether to trust the result of AIbased software or not. Possible solutions for this problem are the integration of probabilistic reasoning and statistical analysis in machine learning (68) as well as quality control (69). Bias and prejudice are well-known problems in medicine (70).…”