Virtual Raters for Reproducible and Objective Assessments in Radiology

Kleesiek, Jens; Petersen, Jens; Döring, Markus; Maier‐Hein, Klaus H.; Köthe, Ullrich; Wick, Wolfgang; Hamprecht, Fred A.; Bendszus, Martin; Biller, Armin

doi:10.1038/srep25007

Cited by 19 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reproducibility of manual segmentations has been investigated previously by others ( Ben Abdallah et al, 2018a , 2016; Bø et al, 2017 ; Cattaneo et al, 2005 ; Gutman et al, 2013 ; Huber et al, 2015 ; Kleesiek et al, 2016 ; Kubben et al, 2010 ; Provenzale et al, 2009 ; Provenzale and Mancini, 2012 ; Sorensen et al, 2001 ; Weltens et al, 2001 ). Most of this work was focused on manual segmentation of preoperative MRI in glioblastoma, although a few of these studies consider longitudinal data ( Huber et al, 2015 ; Kleesiek et al, 2016 ; Kubben et al, 2010 ; Meier et al, 2016 ). Two studies ( Ben Abdallah et al, 2016 ; Huber et al, 2015 ) have addressed the issue of required level of expertise, albeit for preoperative MRI.…”

Section: Introductionmentioning

confidence: 93%

Inter-rater agreement in glioma segmentations on longitudinal MRI

Visser

Müller

Duijn

et al. 2019

NeuroImage: Clinical

View full text Add to dashboard Cite

Background Tumor segmentation of glioma on MRI is a technique to monitor, quantify and report disease progression. Manual MRI segmentation is the gold standard but very labor intensive. At present the quality of this gold standard is not known for different stages of the disease, and prior work has mainly focused on treatment-naive glioblastoma. In this paper we studied the inter-rater agreement of manual MRI segmentation of glioblastoma and WHO grade II-III glioma for novices and experts at three stages of disease. We also studied the impact of inter-observer variation on extent of resection and growth rate. Methods In 20 patients with WHO grade IV glioblastoma and 20 patients with WHO grade II-III glioma (defined as non-glioblastoma) both the enhancing and non-enhancing tumor elements were segmented on MRI, using specialized software, by four novices and four experts before surgery, after surgery and at time of tumor progression. We used the generalized conformity index (GCI) and the intra-class correlation coefficient (ICC) of tumor volume as main outcome measures for inter-rater agreement. Results For glioblastoma, segmentations by experts and novices were comparable. The inter-rater agreement of enhancing tumor elements was excellent before surgery (GCI 0.79, ICC 0.99) poor after surgery (GCI 0.32, ICC 0.92), and good at progression (GCI 0.65, ICC 0.91). For non-glioblastoma, the inter-rater agreement was generally higher between experts than between novices. The inter-rater agreement was excellent between experts before surgery (GCI 0.77, ICC 0.92), was reasonable after surgery (GCI 0.48, ICC 0.84), and good at progression (GCI 0.60, ICC 0.80). The inter-rater agreement was good between novices before surgery (GCI 0.66, ICC 0.73), was poor after surgery (GCI 0.33, ICC 0.55), and poor at progression (GCI 0.36, ICC 0.73). Further analysis showed that the lower inter-rater agreement of segmentation on postoperative MRI could only partly be explained by the smaller volumes and fragmentation of residual tumor. The median interquartile range of extent of resection between raters was 8.3% and of growth rate was 0.22 mm/year. Conclusion Manual tumor segmentations on MRI have reasonable agreement for use in spatial and volumetric analysis. Agreement in spatial overlap is of concern with segmentation after surgery for glioblastoma and with segmentation of non-glioblastoma by non-experts.

show abstract

Section: Introductionmentioning

confidence: 93%

Inter-rater agreement in glioma segmentations on longitudinal MRI

Visser

Müller

Duijn

et al. 2019

NeuroImage: Clinical

View full text Add to dashboard Cite

show abstract

“…The need for a gold standard that quantifies human variability is well‐known and well‐studied in other fields, such as automatic image segmentation, cell counting, or in machine learning (Boccardi et al, ; Entis, Doerga, Barrett, & Dickerson, ; Kleesiek et al, ; Piccinini, Tesei, Paganelli, Zoli, & Bevilacqua, ). For applications such as hippocampi or tumor segmentation, thorough assessments of reproducibility and multiple iterations of manual segmentation protocols already exist (Boccardi et al, ; Frisoni et al, ).…”

Section: Introductionmentioning

confidence: 99%

Tractostorm: The what, why, and how of tractography dissection reproducibility

Rheault

Benedictis

Daducci

et al. 2020

Human Brain Mapping

View full text Add to dashboard Cite

Investigative studies of white matter (WM) brain structures using diffusion MRI (dMRI) tractography frequently require manual WM bundle segmentation, often called “virtual dissection.” Human errors and personal decisions make these manual segmentations hard to reproduce, which have not yet been quantified by the dMRI community. It is our opinion that if the field of dMRI tractography wants to be taken seriously as a widespread clinical tool, it is imperative to harmonize WM bundle segmentations and develop protocols aimed to be used in clinical settings. The EADC‐ADNI Harmonized Hippocampal Protocol achieved such standardization through a series of steps that must be reproduced for every WM bundle. This article is an observation of the problematic. A specific bundle segmentation protocol was used in order to provide a real‐life example, but the contribution of this article is to discuss the need for reproducibility and standardized protocol, as for any measurement tool. This study required the participation of 11 experts and 13 nonexperts in neuroanatomy and “virtual dissection” across various laboratories and hospitals. Intra‐rater agreement (Dice score) was approximately 0.77, while inter‐rater was approximately 0.65. The protocol provided to participants was not necessarily optimal, but its design mimics, in essence, what will be required in future protocols. Reporting tractometry results such as average fractional anisotropy, volume or streamline count of a particular bundle without a sufficient reproducibility score could make the analysis and interpretations more difficult. Coordinated efforts by the diffusion MRI tractography community are needed to quantify and account for reproducibility of WM bundle extraction protocols in this era of open and collaborative science.

show abstract

“…Manual 3D-segmentation is currently a more time-consuming task than solely counting focal lesions. However, there is a development towards automatic tools for combined lesion detection and segmentation [ 23 ], which would reduce the workload for the determination of the SOG.…”

Section: Discussionmentioning

confidence: 99%

Volumetry based biomarker speed of growth: Quantifying the change of total tumor volume in whole-body magnetic resonance imaging over time improves risk stratification of smoldering multiple myeloma patients

et al. 2018

View full text Add to dashboard Cite

The purpose of this study was to improve risk stratification of smoldering multiple myeloma patients, introducing new 3D-volumetry based imaging biomarkers derived from whole-body MRI.Two-hundred twenty whole-body MRIs from 63 patients with smoldering multiple myeloma were retrospectively analyzed and all focal lesions >5mm were manually segmented for volume quantification. The imaging biomarkers total tumor volume, speed of growth (development of the total tumor volume over time), number of focal lesions, development of the number of focal lesions over time and the recent imaging biomarker ‘>1 focal lesion’ of the International Myeloma Working Group were compared, taking 2-year progression rate, sensitivity and false positive rate into account.Speed of growth, using a cutoff of 114mm3/month, was able to isolate a high-risk group with a 2-year progression rate of 82.5%. Additionally, it showed by far the highest sensitivity in this study and in comparison to other biomarkers in the literature, detecting 63.2% of patients who progress within 2 years. Furthermore, its false positive rate (8.7%) was much lower compared to the recent imaging biomarker ‘>1 focal lesion’ of the International Myeloma Working Group.Therefore, speed of growth is the preferable imaging biomarker for risk stratification of smoldering multiple myeloma patients.

show abstract

Virtual Raters for Reproducible and Objective Assessments in Radiology

Cited by 19 publications

References 34 publications

Inter-rater agreement in glioma segmentations on longitudinal MRI

Inter-rater agreement in glioma segmentations on longitudinal MRI

Tractostorm: The what, why, and how of tractography dissection reproducibility

Volumetry based biomarker speed of growth: Quantifying the change of total tumor volume in whole-body magnetic resonance imaging over time improves risk stratification of smoldering multiple myeloma patients

Contact Info

Product

Resources

About