The Validation Index: A New Metric for Validation of Segmentation Algorithms Using Two or More Expert Outlines With Application to Radiotherapy Planning

Abstract

Validation is required to ensure automated segmentation algorithms are suitable for radiotherapy target definition. In the absence of true segmentation, algorithmic segmentation is validated against expert outlining of the region of interest. Multiple experts are used to overcome inter-expert variability. Several approaches have been studied in the literature, but the most appropriate approach to combine the information from multiple expert outlines, to give a single metric for validation, is unclear. None consider a metric that can be tailored to case-specific requirements in radiotherapy planning. Validation index (VI), a new validation metric which uses the experts' level of agreement was developed. A control parameter, , was introduced for the validation of segmentations required for different radiotherapy scenarios: for targets close to organs-at-risk and for difficult to discern targets, where large variation between experts is expected. VI was evaluated using two simulated idealised cases and data from two clinical studies. VI was compared with the commonly used Dice similarity coefficient (DSC pair-wise ) and found to be more sensitive than the DSC pair-wise to the changes in agreement between experts. VI was shown to be adaptable to specific radiotherapy planning scenarios.

The feasibility of using citizens to segment anatomy from medical images: Accuracy and motivation

“…The mean expert agreement Dice score of 0.96 was similar to previous studies that have found a Dice scores of 0.9 for the aorta [12], 0.81 for white matter in the brain [13], and 0.80, 0.92 and 0.88 for breast, brain tissue and brain tumour respectively [14]. The better agreement in the anterior regions of the spine (the vertebral body) compared to the posterior regions (posterior elements) (Fig 4) is consistent with the findings of Deeley, Chen (11) that agreement tends to be higher when segmenting large round structures and lower when segmenting small thin structures.…”

Designing image segmentation studies: Statistical power, sample size and reference standard quality

“…Newer metrics allow comparisons to multiple reference standards (e.g. the validation index ( Juneja et al., 2013 )) or comparisons that consider application specific utility (e.g. accuracy of quantitative measurements in segmented ROIs ( Jha et al., 2012 )).…”

Assessment of fully-automated atlas-based segmentation of novel oral mucosal surface organ-at-risk

“…The VI is a recently designed measure, for geometric comparison of automated and multiple manually contoured structures, that attempts to account for uncertainties in the manual contouring [26].…”