Annotation quality vs. quantity for deep-learned medical image segmentation

Wesemeyer, Tim; Jauer, Malte-Levin; Deserno, Thomas M.

doi:10.1117/12.2582226

Cited by 9 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 37 Contrary to the previous studies, Wesemeyer et al. 38 demonstrated a trade-off between quality and quantity for deep learning segmentation performance. As we illustrate in our study, expert consistency may be better than nonexpert consistency for some radiotherapy-related ROIs, particularly for H&N imaging.…”

Section: Discussionmentioning

confidence: 85%

“…36 Additionally, the authors show no statistical difference between the variability of experts vs. novices in several of the annotations of interest, results which are echoed in a recent study investigating crowdsourcing for liver tumor segmentation where the quality of annotations was not statistically significantly different in four distinct groups. 37 Contrary to the previous studies, Wesemeyer et al 38 demonstrated a trade-off between quality and quantity for deep learning segmentation performance. As we illustrate in our study, expert consistency may be better than nonexpert consistency for some radiotherapy-related ROIs, particularly for H&N imaging.…”

Section: Nonexpert Staple Bootstrap Experimentsmentioning

confidence: 85%

See 1 more Smart Citation

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

Lin

Wahid

Nelms³

et al. 2023

J. Med. Imag.

View full text Add to dashboard Cite

. Purpose Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach Participants who contoured region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. ROIs were evaluated against contours using Dice similarity coefficient (DSC). The expert interobserver DSC ( ) was calculated as an acceptability threshold between and . To determine the number of nonexperts required to match the for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the . Results For all cases, the DSC values for versus were higher than comparator expert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H&N, 3 and 5 for GYN, and 3 for GI. Conclusions Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

show abstract

Section: Discussionmentioning

confidence: 85%

Section: Nonexpert Staple Bootstrap Experimentsmentioning

confidence: 85%

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

Lin

Wahid

Nelms³

et al. 2023

J. Med. Imag.

View full text Add to dashboard Cite

show abstract

“…Additionally, the authors show average DSC for expert-expert and novice-expert segmentations show that no statistical difference is found between the variability of annotators in several of the annotations of interest, results which are echoed in a recent study investigating crowdsourcing for liver tumor segmentation where the quality of annotations was not statistically significantly different four distinct groups [35]. Contrary to the previous studies, Wesemeyer et al demonstrate a tradeoff between quality and quantity for deep learning segmentation performance [36]. As we demonstrate in our study, expert consistency may be better than non-expert consistency for some radiotherapy-related ROIs, particularly for H&N imaging.…”

Section: Discussionmentioning

confidence: 98%

“E Pluribus Unum”: Prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology (C3RO) Crowdsourced Initiative for Multi-Observer Segmentation

Lin

Wahid

Nelms³

et al. 2022

Preprint

View full text Add to dashboard Cite

OBJECTIVE: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. A challenge in artificial intelligence (AI) development is the paucity of multi-expert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple non-experts could meet or exceed recognized expert agreement. MATERIALS AND METHODS: Participants who contoured ≥1 region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) challenge were identified as a non-expert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLEnon-expert ROIs were evaluated against STAPLEexpert contours using Dice Similarity Coefficient (DSC). The expert interobserver DSC (IODSCexpert) was calculated as an acceptability threshold between STAPLEnon-expert and STAPLEexpert. To determine the number of non-experts required to match the IODSCexpert for each ROI, a single consensus contour was generated using variable numbers of non-experts and then compared to the IODSCexpert. RESULTS: For all cases, the DSC for STAPLEnon-expert versus STAPLEexpert were higher than comparator expert IODSCexpert for most ROIs. The minimum number of non-expert segmentations needed for a consensus ROI to achieve IODSCexpert acceptability criteria ranged between 2-4 for breast, 3-5 for sarcoma, 3-5 for H&N, 3-5 for GYN ROIs, and 3 for GI ROIs. DISCUSSION AND CONCLUSION: Multiple non-expert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. 5 non-experts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting non-expert segmentations as feasible cost-effective AI inputs.

show abstract

“…However, for auto-segmentation algorithms to be clinically useful, their input data (training data) should reflect high-quality "gold-standard" annotations. While research has been performed on the impact of interobserver variability and segmentation quality for auto-segmentation training [8][9][10][11] , it remains unclear how "gold-standard" segmentations should be defined and generated. One common approach, consensus segmentation generation, seeks to crowdsource multiple segmentations from different annotators to generate a high-quality ground-truth segmentation.…”

Section: Background and Summarymentioning

confidence: 99%

Large scale crowdsourced radiotherapy segmentations across a variety of cancer anatomic sites

et al. 2023

View full text Add to dashboard Cite

Clinician generated segmentation of tumor and healthy tissue regions of interest (ROIs) on medical images is crucial for radiotherapy. However, interobserver segmentation variability has long been considered a significant detriment to the implementation of high-quality and consistent radiotherapy dose delivery. This has prompted the increasing development of automated segmentation approaches. However, extant segmentation datasets typically only provide segmentations generated by a limited number of annotators with varying, and often unspecified, levels of expertise. In this data descriptor, numerous clinician annotators manually generated segmentations for ROIs on computed tomography images across a variety of cancer sites (breast, sarcoma, head and neck, gynecologic, gastrointestinal; one patient per cancer site) for the Contouring Collaborative for Consensus in Radiation Oncology challenge. In total, over 200 annotators (experts and non-experts) contributed using a standardized annotation platform (ProKnow). Subsequently, we converted Digital Imaging and Communications in Medicine data into Neuroimaging Informatics Technology Initiative format with standardized nomenclature for ease of use. In addition, we generated consensus segmentations for experts and non-experts using the Simultaneous Truth and Performance Level Estimation method. These standardized, structured, and easily accessible data are a valuable resource for systematically studying variability in segmentation applications.

show abstract

Annotation quality vs. quantity for deep-learned medical image segmentation

Cited by 9 publications

References 0 publications

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation

“E Pluribus Unum”: Prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology (C3RO) Crowdsourced Initiative for Multi-Observer Segmentation

Large scale crowdsourced radiotherapy segmentations across a variety of cancer anatomic sites

Contact Info

Product

Resources

About