Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence

Maier-Hein, Lena; Roß, Tobias; Gröhl, Janek; Glocker, Ben; Bodenstedt, Sebastian; Stock, Christian; Heim, Eric; Goetz, Michael; Wirkert, Sebastian J.; Kenngott, Hannes; Speidel, Stefanie; Maier‐Hein, Klaus H.

doi:10.1007/978-3-319-46723-8_71

Cited by 36 publications

(34 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, methods for crowd-algorithm collaboration could be investigated (e.g. [65], [66]) to further reduce annotation costs. In conclusion, we believe that our method has a great potential for use in large-scale low-cost data annotation.…”

Section: Discussionmentioning

confidence: 99%

Clickstream Analysis for Crowd-Based Object Segmentation with Confidence

Heim

Seitel

Andrulis

et al. 2018

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Abstract-With the rapidly increasing interest in machine learning based solutions for automatic image annotation, the availability of reference annotations for algorithm training is one of the major bottlenecks in the field. Crowdsourcing has evolved as a valuable option for low-cost and large-scale data annotation; however, quality control remains a major issue which needs to be addressed. To our knowledge, we are the first to analyze the annotation process to improve crowd-sourced image segmentation. Our method involves training a regressor to estimate the quality of a segmentation from the annotator's clickstream data. The quality estimation can be used to identify spam and weight individual annotations by their (estimated) quality when merging multiple segmentations of one image. Using a total of 29,000 crowd annotations performed on publicly available data of different object classes, we show that (1) our method is highly accurate in estimating the segmentation quality based on clickstream data, (2) outperforms state-of-the-art methods for merging multiple annotations. As the regressor does not need to be trained on the object class that it is applied to it can be regarded as a low-cost option for quality control and confidence analysis in the context of crowd-based image annotation.

show abstract

Section: Discussionmentioning

confidence: 99%

Clickstream Analysis for Crowd-Based Object Segmentation with Confidence

Heim

Seitel

Andrulis

et al. 2018

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…This method has been applied to brain lesion segmentation [6], [22] and surgical tool segmentation [10]. Two example applications of the uncertainty metric explored in this study are; 1) prediction of segmentation accuracy without using the ground truth similar to the goal of Valindria et al [35] and, 2) the active-learning framework [19], [39] for the reduction of manual annotation costs.…”

Section: A Related Workmentioning

confidence: 99%

Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling

Hiasa

Otake

Takao

et al. 2020

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

We propose a method for automatic segmentation of individual muscles from a clinical CT. The method uses Bayesian convolutional neural networks with the U-Net architecture, using Monte Carlo dropout that infers an uncertainty metric in addition to the segmentation label. We evaluated the performance of the proposed method using two data sets: 20 fully annotated CTs of the hip and thigh regions and 18 partially annotated CTs that are publicly available from The Cancer Imaging Archive (TCIA) database. The experiments showed a Dice coefficient (DC) of 0.891±0.016 (mean±std) and an average symmetric surface distance (ASD) of 0.994±0.230 mm over 19 muscles in the set of 20 CTs. These results were statistically significant improvements compared to the state-of-the-art hierarchical multi-atlas method which resulted in 0.845±0.031 DC and 1.556±0.444 mm ASD. We evaluated validity of the uncertainty metric in the multi-class organ segmentation problem and demonstrated a correlation between the pixels with high uncertainty and the segmentation failure. One application of the uncertainty metric in activelearning is demonstrated, and the proposed query pixel selection method considerably reduced the manual annotation cost for expanding the training data set. The proposed method allows an accurate patient-specific analysis of individual muscle shapes in a clinical routine. This would open up various applications including personalization of biomechanical simulation and quantitative evaluation of muscle atrophy.

show abstract

“…The model has been applied to several hundreds liver tumor patients, and is currently being extended to applications in renal surgery, including intraoperative process models based on LapOntoSPM. In this context, new methods for large-scale medical data annotation based on crowdsourcing have been developed [25,40,41]. An implementation of the system is publicly available [69].…”

Section: Ontology Development In Heidelbergmentioning

confidence: 99%