Combining natural and artificial intelligence for robust automatic anatomy segmentation: Application in neck and thorax auto‐contouring

Abstract

Background Automatic segmentation of 3D objects in computed tomography (CT) is challenging. Current methods, based mainly on artificial intelligence (AI) and end‐to‐end deep learning (DL) networks, are weak in garnering high‐level anatomic information, which leads to compromised efficiency and robustness. This can be overcome by incorporating natural intelligence (NI) into AI methods via computational models of human anatomic knowledge. Purpose We formulate a hybrid intelligence (HI) approach that integrates the complementary strengths of NI and AI for organ segmentation in CT images and illustrate performance in the application of radiation therapy (RT) planning via multisite clinical evaluation. Methods The system employs five modules: (i) body region recognition, which automatically trims a given image to a precisely defined target body region; (ii) NI‐based automatic anatomy recognition object recognition (AAR‐R), which performs object recognition in the trimmed image without DL and outputs a localized fuzzy model for each object; (iii) DL‐based recognition (DL‐R), which refines the coarse recognition results of AAR‐R and outputs a stack of 2D bounding boxes (BBs) for each object; (iv) model morphing (MM), which deforms the AAR‐R fuzzy model of each object guided by the BBs output by DL‐R; and (v) DL‐based delineation (DL‐D), which employs the object containment information provided by MM to delineate each object. NI from (ii), AI from (i), (iii), and (v), and their combination from (iv) facilitate the HI system. Results The HI system was tested on 26 organs in neck and thorax body regions on CT images obtained prospectively from 464 patients in a study involving four RT centers. Data sets from one separate independent institution involving 125 patients were employed in training/model building for each of the two body regions, whereas 104 and 110 data sets from the 4 RT centers were utilized for testing on neck and thorax, respectively. In the testing data sets, 83% of the images had limitations such as streak artifacts, poor contrast, shape distortion, pathology, or implants. The contours output by the HI system were compared to contours drawn in clinical practice at the four RT centers by utilizing an independently established ground‐truth set of contours as reference. Three sets of measures were employed: accuracy via Dice coefficient (DC) and Hausdorff boundary distance (HD), subjective clinical acceptability via a blinded reader study, and efficiency by measuring human time saved in contouring by the HI system. Overall, the HI system achieved a mean DC of 0.78 and 0.87 and a mean HD of 2.22 and 4.53 mm for neck and thorax, respectively. It significantly outperformed clinical contouring in accuracy and saved overall 70% of human time over clinical contouring time, whereas acceptability scores varied significantly from site to site for both auto‐contours and clinically drawn contours. Conclusions The HI system is observed to behave like an expert human in robustness in the contouring task but vast...

HaN‐Seg: The head and neck organ‐at‐risk CT and MR segmentation dataset

“…On the other hand, from the perspective of OAR annotation, it contains reference segmentations of the most complete set of 30 OARs in the HaN region. Finally, from the perspective of auto‐segmentation performance, the baseline results obtained by an off‐the‐shelf DL solution (i.e., nnU‐Net) are inferior to the results reported by existing CT‐only DL methods 9 but still comparable to some recent state‐of‐the‐art DL methods 44 . With the release of the HaN‐Seg dataset and deployment of the accompanying HaN‐Seg challenge, our aim is, therefore, to test the hypothesis that the accuracy and reliability of OAR segmentation can be improved by exploiting the fused information from both CT and MR images, with the objective to design, develop, and evaluate novel auto‐segmentation algorithms that rely on robust and accurate registration algorithms and can be benchmarked on a common dataset.…”

“…Finally, from the perspective of auto-segmentation performance, the baseline results obtained by an off -the-shelf DL solution (i.e., nnU-Net) are inferior to the results reported by existing CT-only DL methods 9 but still comparable to some recent state-ofthe-art DL methods. 44 With the release of the HaN-Seg dataset and deployment of the accompanying HaN-Seg challenge, our aim is, therefore, to test the hypothesis that the accuracy and reliability of OAR segmentation can be improved by exploiting the fused information from both CT and MR images, with the objective to design, develop, and evaluate novel auto-segmentation algorithms that rely on robust and accurate registration algorithms and can be benchmarked on a common dataset. Considering the recent growth in the application of AI, especially DL, 45 the devised HaN-Seg dataset has also the potential to contribute to a more objective and trustful reporting of research outcomes.…”

Mendability Index: a new metric for estimating the effort required for manually editing auto-segmentations of objects of interest

“…Therefore, 𝛿 and 𝜀 may need to be specifically set or optimized for each object of interest. In this work, we decide 𝛿 based on the differences between two sets of manual segmentation results for the same input images from our previous study [10]. Specifically, we calculate the HD between two manual segmentation masks for the same image.…”

“…To validate how well the proposed MI metric can be used to estimate the effort required for manually editing autosegmentation of objects of interest, we collected CT data and annotation labels for 6 objects of interest from 3 institutes [10]. There are 64 cases of left submandibular gland (LSmG), 87 cases of mandible (Mnd), 89 cases of orohypopharyngeal constrictor muscles (OHPh), 87 cases of cervical trachea (N-Tr), 89 cases of thoracic esophagus (T-Es), and 89 cases of heart (Hrt).…”

“…However, different from the segmentation task for general natural images, how to quantitatively evaluate the performance of a medical image segmentation method following clinical demands has not been well studied. Although Dice Coefficient (DC) [5] and Hausdorff Distance (HD) [6] are two popular metrics for segmentation evaluation considering area overlap and boundary distance, respectively, how well these scores reflect the practical value of automatic segmentation results in clinical situations is not fully studied [7,10,11]. Two automatic segmentation results with large false positive areas and relatively small false positive areas may have the same HD or DC [8,11] for the same object or different objects, whereas the radiologist, radiation oncologist, or dosimetrist may spend very different amounts of time and effort to individually correct the segmentation masks, depending on the clinical indication and the degree of accuracy required for the segmentation task at hand.…”