Novel Autosegmentation Spatial Similarity Metrics Capture the Time Required to Correct Segmentations Better Than Traditional Metrics in a Thoracic Cavity Segmentation Workflow

Abstract

Automated segmentation templates can save clinicians time compared to de novo segmentation but may still take substantial time to review and correct. It has not been thoroughly investigated which automated segmentation-corrected segmentation similarity metrics best predict clinician correction time. Bilateral thoracic cavity volumes in 329 CT scans were segmented by a UNet-inspired deep learning segmentation tool and subsequently corrected by a fourth-year medical student. Eight spatial similarity metrics were calculated between the automated and corrected segmentations and associated with correction times using Spearman’s rank correlation coefficients. Nine clinical variables were also associated with metrics and correction times using Spearman’s rank correlation coefficients or Mann–Whitney U tests. The added path length, false negative path length, and surface Dice similarity coefficient correlated better with correction time than traditional metrics, including the popular volumetric Dice similarity coefficient (respectively ρ = 0.69, ρ = 0.65, ρ = − 0.48 versus ρ = − 0.25; correlation p values < 0.001). Clinical variables poorly represented in the autosegmentation tool’s training data were often associated with decreased accuracy but not necessarily with prolonged correction time. Metrics used to develop and evaluate autosegmentation tools should correlate with clinical time saved. To our knowledge, this is only the second investigation of which metrics correlate with time saved. Validation of our findings is indicated in other anatomic sites and clinical workflows. Novel spatial similarity metrics may be preferable to traditional metrics for developing and evaluating autosegmentation tools that are intended to save clinicians time.

RootPainter3D: Interactive‐machine‐learning enables rapid and accurate contouring for radiotherapy

“…Despite editing auto-contours being identified as a barrier to the adoption of automatic methods, 47 our results, in alignment with previous studies, [51][52][53] demonstrate that correcting auto-contours saves time compared to a standard clinical workflow. In prior work, timesavings have been demonstrated, even when the predicted structures are of particularly low quality, with a dice score as low as 0.46.…”

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

“…For our analysis, we focused on the Dice similarity coefficient (DSC), a well-established volume-based metric for segmentation studies, and the surface DSC (SDSC), a newer surface distance metric that has been shown to be germane to potentially improving radiation oncology workflows, particularly for time savings. 27 , 28 Metrics were calculated using the surface-distances Python package 29 and in-house Python code. SDSC was calculated based on ROI-specific thresholds determined by measuring the median pairwise mean surface distance of all expert segmentations for that ROI as suggested in the literature; 29 tolerance values, required parameters for SDSC calculation, used for each ROI are shown in Table S2 in the Supplementary Material .…”

Machine learning-based detection of aberrant deep learning segmentations of target and organs at risk for prostate radiotherapy using a secondary segmentation algorithm

“…Therefore, two recently developed evaluation metrics, namely surface DSC and APL were also included in the calculated metrics. These metrics have been shown to be better indicators for the clinical delineation time saved (Vaassen et al 2020, Kiser et al 2021. They may provide additional objectively quantifiable surrogates for assessing time-saving and clinical applicability and quality of automatically generated contours in the delineation process (Gooding et al 2018).…”