Learning Calibrated Medical Image Segmentation via Multi-rater Agreement Modeling

Ji, Wei; Yu, Shuang; Wu, Junde; Ma, Kai; Bian, Cheng; Bi, Qi; Li, Jingjing; Liu, Hanruo; Cheng, Li; Zheng, Yefeng

doi:10.1109/cvpr46437.2021.01216

Cited by 110 publications

(51 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead, in many cases, subjective labels from multiple experts are available. This raises the recent research attention [1,22,28,29,31,45] on multi-rater problem. Many works leveraged multi-rater annotations to attain calibrated result [28,29].…”

Section: Related Workmentioning

confidence: 99%

“…This raises the recent research attention [1,22,28,29,31,45] on multi-rater problem. Many works leveraged multi-rater annotations to attain calibrated result [28,29]. However, they still took majority vote as ground-truth in model evaluation, ignoring the discrepancy between majority vote and the potential gold standard.…”

Section: Related Workmentioning

confidence: 99%

“…To demonstrate the advantage of the proposed DiFF, we compare our method with the state-of-the-art (SOTA) methods proposed for the multi-rater problem, including UNet based on majority vote (MV-UNet) [42], LFC (LFC-UNet) [41], STAPLE (STAPLE-UNet) [52], multi-head strategy (MH-UNet) [22] and MRNet [29]. We compare AUC of the masks estimated by proposed DiFF with SOTA methods in Figure 5.…”

Section: Comparisons With State-of-the-artsmentioning

confidence: 99%

“…Clinically, it is often necessary to synthesize the opinions of several different clinical experts as a reference. In the practice of medical image processing, prior work [3,29,56] called it 'multi-rater problem', which means that each instance of the collected dataset is annotated by several different raters. One multi-rater example of optic cup segmentation is shown in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Opinions Vary? Diagnosis First!

Wu¹,

Fang²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In medical image segmentation, images are usually annotated by several different clinical experts. This clinical routine helps to mitigate the personal bias. However, Computer Vision models often assume there has a unique ground-truth for each of the instance. This research gap between Computer Vision and medical routine is commonly existed but less explored by the current research. In this paper, we try to answer the following two questions: 1. How to learn an optimal combination of the multiple segmentation labels? and 2. How to estimate this segmentation mask from the raw image? We note that in clinical practice, the image segmentation mask usually exists as an auxiliary information for disease diagnosis. Adhering to this mindset, we propose a framework taking the diagnosis result as the gold standard, to estimate the segmentation mask upon the multi-rater segmentation labels, named DiFF (Diagnosis First segmentation Framework). DiFF is implemented by two novelty techniques. First, DFSim (Diagnosis First Simulation of gold label) is learned as an optimal combination of multi-rater segmentation labels for the disease diagnosis. Then, toward estimating DFSim mask from the raw image, we further propose T&G Module (Take and Give Module) to instill the diagnosis knowledge into the segmentation network. The experiments show that compared with commonly used majority vote, the proposed DiFF is able to segment the masks with 6% improvement on diagnosis AUC score, which also outperforms various state-of-the-art multi-rater methods by a large margin.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Comparisons With State-of-the-artsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Opinions Vary? Diagnosis First!

Wu¹,

Fang²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This increased ability to encode the inter-rater variability is probably due to the fact that SoftSeg facilitates the propagation of soft labels throughout the training scheme: (1) no binarization of the input labels, (2) a loss function which does not penalize uncertain predictions, and (3) an activation function which does not enforce binary outputs. Considered with equivalent expertise in this work, future studies could account for the different expertise across raters, for instance by modulating the training scheme with FiLM layers , or by the use of expertise-aware inferring module Ji et al (2021).…”

Section: When Using the Labels Through The Training Pipelinementioning

confidence: 99%

Label fusion and training methods for reliable representation of inter-rater uncertainty

Lemay¹,

Gros²,

Cohen‐Adad³

2022

Preprint

View full text Add to dashboard Cite

Medical tasks are prone to inter-rater variability due to multiple factors such as image quality, professional experience and training, or guideline clarity. Training deep learning networks with annotations from multiple raters is a common practice that mitigates the model's bias towards a single expert. Reliable models generating calibrated outputs and reflecting the inter-rater disagreement are key to the integration of artificial intelligence in clinical practice. Various methods exist to take into account different expert labels. We focus on comparing three label fusion methods: STAPLE, average of the rater's segmentation, and random sampling of each rater's segmentation during training. Each label fusion method is studied using both the conventional training framework and the recently published SoftSeg framework that limits information loss by treating the segmentation task as a regression. Our results, across 10 data splittings on two public datasets (spinal cord gray matter challenge, and multiple sclerosis brain lesion segmentation), indicate that SoftSeg models, regardless of the ground truth fusion method, had better calibration and preservation of the inter-rater rater variability compared with their conventional counterparts without impacting the segmentation performance. Conventional models, i.e., trained with a Dice loss, with binary inputs, and sigmoid/softmax final activate, were overconfident and underestimated the uncertainty associated with inter-rater variability. Conversely, fusing labels by averaging with the SoftSeg framework led to underconfident outputs and overestimation of the rater disagreement. In terms of segmentation performance, the best label fusion method was different for the two datasets studied, indicating this parameter might be task-dependent. However, SoftSeg had segmentation performance systematically superior or equal to the conventionally trained models and had the best calibration and preservation of the inter-rater variability. Our code is available at https://ivadomed.org.

show abstract

A novel adaptive cubic quasi‐Newton optimizer for deep learning based medical image analysis tasks, validated on detection of COVID‐19 and segmentation for COVID‐19 lung infection, liver tumor, and optic disc/cup

et al. 2022

View full text Add to dashboard Cite

Background Most of existing deep learning research in medical image analysis is focused on networks with stronger performance. These networks have achieved success, while their architectures are complex and even contain massive parameters ranging from thousands to millions in numbers. The nature of high dimension and nonconvex makes it easy to train a suboptimal model through the popular stochastic first‐order optimizers, which only use gradient information. Purpose Our purpose is to design an adaptive cubic quasi‐Newton optimizer, which could help to escape from suboptimal solution and improve the performance of deep neural networks on four medical image analysis tasks including: detection of COVID‐19, COVID‐19 lung infection segmentation, liver tumor segmentation, optic disc/cup segmentation. Methods In this work, we introduce a novel adaptive cubic quasi‐Newton optimizer with high‐order moment (termed ACQN‐H) for medical image analysis. The optimizer dynamically captures the curvature of the loss function by diagonally approximated Hessian and the norm of difference between previous two estimates, which helps to escape from saddle points more efficiently. In addition, to reduce the variance introduced by the stochastic nature of the problem, ACQN‐H hires high‐order moment through exponential moving average on iteratively calculated approximated Hessian matrix. Extensive experiments are performed to access the performance of ACQN‐H. These include detection of COVID‐19 using COVID‐Net on dataset COVID‐chestxray, which contains 16 565 training samples and 1841 test samples; COVID‐19 lung infection segmentation using Inf‐Net on COVID‐CT, which contains 45, 5, and 5 computer tomography (CT) images for training, validation, and testing, respectively; liver tumor segmentation using ResUNet on LiTS2017, which consists of 50 622 abdominal scan images for training and 26 608 images for testing; optic disc/cup segmentation using MRNet on RIGA, which has 655 color fundus images for training and 95 for testing. The results are compared with commonly used stochastic first‐order optimizers such as Adam, SGD, and AdaBound, and recently proposed stochastic quasi‐Newton optimizer Apollo. In task detection of COVID‐19, we use classification accuracy as the evaluation metric. For the other three medical image segmentation tasks, seven commonly used evaluation metrics are utilized, that is, Dice, structure measure, enhanced‐alignment measure (EM), mean absolute error (MAE), intersection over union (IoU), true positive rate (TPR), and true negative rate. Results Experiments on four tasks show that ACQN‐H achieves improvements over other stochastic optimizers: (1) comparing with AdaBound, ACQN‐H achieves 0.49%, 0.11%, and 0.70% higher accuracy on the COVID‐chestxray dataset using network COVID‐Net with VGG16, ResNet50 and DenseNet121 as backbones, respectively; (2) ACQN‐H has the best scores in terms of evalua...

show abstract

Learning Calibrated Medical Image Segmentation via Multi-rater Agreement Modeling

Cited by 110 publications

References 39 publications

Opinions Vary? Diagnosis First!

Opinions Vary? Diagnosis First!

Label fusion and training methods for reliable representation of inter-rater uncertainty

A novel adaptive cubic quasi‐Newton optimizer for deep learning based medical image analysis tasks, validated on detection of COVID‐19 and segmentation for COVID‐19 lung infection, liver tumor, and optic disc/cup

Contact Info

Product

Resources

About