Comparative Validation of Graphical Models for Learning Tumor Segmentations from Noisy Manual Annotations

Kaster, Frederik O.; Menze, Bjoern; Weber, Marc‐André; Hamprecht, Fred A.

doi:10.1007/978-3-642-18421-5_8

Cited by 11 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optimizing the ensemble prediction by balancing variability reduction (fuse many predictors) and bias removal (fuse a few selected only) can be done on a test set representing the overall population, or for the individual image volume when partial annotation is available—for example from the limited user interaction mentioned above. Statistical methods that estimate and weight the performance of individual contributions—for example, based on appropriate multi-class extensions of STAPLE [69] and related probabilistic models [19], [84]—may also be used to trade bias and variance in an optimal fashion.…”

Section: Discussionmentioning

confidence: 99%

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Menze

Jakab

Bauer

et al. 2015

IEEE Trans. Med. Imaging

Self Cite

4,428

2,856

View full text Add to dashboard Cite

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients—manually annotated by up to four raters—and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%–85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

show abstract

Section: Discussionmentioning

confidence: 99%

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Menze

Jakab

Bauer

et al. 2015

IEEE Trans. Med. Imaging

Self Cite

4,428

2,856

View full text Add to dashboard Cite

show abstract

“…Some methods in this category improve standard classifiers such as support vector machines, decision trees, and neural networks by proposing novel training procedures that are more robust to label noise (Khardon and Wachman, 2007;Lin et al, 2004). Alternatively, different forms of probabilistic models have been used to model the label noise and thereby improve various classifiers (Kaster et al, 2010;Kim and Ghahramani, 2006).…”

mentioning

confidence: 99%

Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis

Karimi

Dou

Warfield

et al. 2020

Medical Image Analysis

452

234

View full text Add to dashboard Cite

Supervised training of deep learning models requires large labeled datasets. There is a growing interest in obtaining such datasets for medical image analysis applications. However, the impact of label noise has not received sufficient attention. Recent studies have shown that label noise can significantly impact the performance of deep learning models in many machine learning and computer vision applications. This is especially concerning for medical applications, where datasets are typically small, labeling requires domain expertise and suffers from high inter-and intra-observer variability, and erroneous predictions may influence decisions that directly impact human health. In this paper, we first review the state-of-the-art in handling label noise in deep learning. Then, we review studies that have dealt with label noise in deep learning for medical image analysis. Our review shows that recent progress on handling label noise in deep learning has gone largely unnoticed by the medical image analysis community. To help achieve a better understanding of the extent of the problem and its potential remedies, we conducted experiments with three medical imaging datasets with different types of label noise, where we investigated several existing strategies and developed new methods to combat the negative effect of label noise. Based on the results of these experiments and our review of the literature, we have made recommendations on methods that can be used to alleviate the effects of different types of label noise on deep models trained for medical image analysis. We hope that this article helps the medical image analysis researchers and developers in choosing and devising new techniques that effectively handle label noise in deep learning.

show abstract

“…[11,12] detect label noise based on their change of the classification of other samples in a leave-one-out framework. Furthermore, probabilistic models with a novel form have been used and also improve the performance of the classifier [13,14].…”

Section: Related Workmentioning

confidence: 99%

Leveraging Expert Knowledge for Label Noise Mitigation in Machine Learning

et al. 2021

View full text Add to dashboard Cite

In training-based Machine Learning applications, the training data are frequently labeled by non-experts and expose substantial label noise which greatly alters the training models. In this work, a novel method for reducing the effect of label noise is introduced. The rules are created from expert knowledge to identify the incorrect non-expert training data. Using the gradient descent algorithm, the violating data samples are weighted less to mitigate their effects during model training. The proposed method is applied to the image classification problem using Manga109 and CIFAR-10 dataset. The experiments show that when the noise level is up to 50% our proposed method significantly increases the accuracy of the model compared to conventional learning methods.

show abstract

Comparative Validation of Graphical Models for Learning Tumor Segmentations from Noisy Manual Annotations

Cited by 11 publications

References 11 publications

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)

Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis

Leveraging Expert Knowledge for Label Noise Mitigation in Machine Learning

Contact Info

Product

Resources

About