Interobserver variability in quality assessment of magnetic resonance images

Obuchowicz, Rafał; Oszust, Mariusz; Piórkowski, Adam

doi:10.1186/s12880-020-00505-z

Cited by 17 publications

(10 citation statements)

References 43 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meanwhile, the quality of MR images is mainly dependent on the scanning parameters, which are usually changed across different institutions or even varied in each MR scanner in the same center [19]. Therefore, the subjective bias of assessment and segmentation of lesion areas in the MR images seems inevitable, even between experienced radiologists [20]. This, in turn will have a substantial impact on the training and validation of the segmentation neural networks.…”

Section: Discussionmentioning

confidence: 99%

Deep learning-based high-accuracy detection for lumbar and cervical degenerative disease on T2-weighted MR images

et al. 2023

View full text Add to dashboard Cite

Purpose To develop and validate a deep learning (DL) model for detecting lumbar degenerative disease in both sagittal and axial views of T2-weighted MRI and evaluate its generalized performance in detecting cervical degenerative disease. Methods T2-weighted MRI scans of 804 patients with symptoms of lumbar degenerative disease were retrospectively collected from three hospitals. The training dataset (n = 456) and internal validation dataset (n = 134) were randomly selected from the center I. Two external validation datasets comprising 100 and 114 patients were from center II and center III, respectively. A DL model based on 3D ResNet18 and transformer architecture was proposed to detect lumbar degenerative disease. In addition, a cervical MR image dataset comprising 200 patients from an independent hospital was used to evaluate the generalized performance of the DL model. The diagnostic performance was assessed by the free-response receiver operating characteristic (fROC) curve and precision–recall (PR) curve. Precision, recall, and F1-score were used to measure the DL model. Results A total of 2497 three-dimension retrogression annotations were labeled for training (n = 1157) and multicenter validation (n = 1340). The DL model showed excellent detection efficiency in the internal validation dataset, with F1-score achieving 0.971 and 0.903 on the sagittal and axial MR images, respectively. Good performance was also observed in the external validation dataset I (F1-score, 0.768 on sagittal MR images and 0.837 on axial MR images) and external validation dataset II (F1-score, 0.787 on sagittal MR images and 0.770 on axial MR images). Furthermore, the robustness of the DL model was demonstrated via transfer learning and generalized performance evaluation on the external cervical dataset, with the F1-score yielding 0.931 and 0.919 on the sagittal and axial MR images, respectively. Conclusion The proposed DL model can automatically detect lumbar and cervical degenerative disease on T2-weighted MR images with good performance, robustness, and feasibility in clinical practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep learning-based high-accuracy detection for lumbar and cervical degenerative disease on T2-weighted MR images

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Some datasets present labels that can support supervised learning strategies for classification tasks (often related to the patient clinical diagnosis), or segmentation tasks (brain structures or brain abnormalities related to a specific pathology). Manual labeling, however, can be time-consuming, expensive, prone to errors, and may present with unacceptable intra- and inter-expert variation (Obuchowicz et al, 2020 ). There are different approaches to overcome these challenges, such as using consensus methods, like simultaneous truth and performance level estimation (STAPLE, Warfield et al, 2004 ), and unsupervised approaches (Shen, 2013 ; Souza et al, 2018 ).…”

Section: Data Accessmentioning

confidence: 99%

“…The challenges of batch effects may be further exacerbated in the era of DL as new, large, multi-site, heterogeneous datasets are created by combining (or aggregating) multiple previously acquired datasets (Bento et al, 2019 ; Lee et al, 2020 ; Zlochower et al, 2020 ). The impact of data variability must be first understood to ensure that these models are answering the proposed research question, not only in the presence of inherent image variability, but also after addressing potential misclassification or misdiagnosis attributable to this variability (Obuchowicz et al, 2020 ). Large and heterogeneous datasets are key requirements for model development to avoid limiting the model to a specific cohort.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Bento

Fantini

Park

et al. 2022

Front. Neuroinform.

View full text Add to dashboard Cite

Large, multi-site, heterogeneous brain imaging datasets are increasingly required for the training, validation, and testing of advanced deep learning (DL)-based automated tools, including structural magnetic resonance (MR) image-based diagnostic and treatment monitoring approaches. When assembling a number of smaller datasets to form a larger dataset, understanding the underlying variability between different acquisition and processing protocols across the aggregated dataset (termed “batch effects”) is critical. The presence of variation in the training dataset is important as it more closely reflects the true underlying data distribution and, thus, may enhance the overall generalizability of the tool. However, the impact of batch effects must be carefully evaluated in order to avoid undesirable effects that, for example, may reduce performance measures. Batch effects can result from many sources, including differences in acquisition equipment, imaging technique and parameters, as well as applied processing methodologies. Their impact, both beneficial and adversarial, must be considered when developing tools to ensure that their outputs are related to the proposed clinical or research question (i.e., actual disease-related or pathological changes) and are not simply due to the peculiarities of underlying batch effects in the aggregated dataset. We reviewed applications of DL in structural brain MR imaging that aggregated images from neuroimaging datasets, typically acquired at multiple sites. We examined datasets containing both healthy control participants and patients that were acquired using varying acquisition protocols. First, we discussed issues around Data Access and enumerated the key characteristics of some commonly used publicly available brain datasets. Then we reviewed methods for correcting batch effects by exploring the two main classes of approaches: Data Harmonization that uses data standardization, quality control protocols or other similar algorithms and procedures to explicitly understand and minimize unwanted batch effects; and Domain Adaptation that develops DL tools that implicitly handle the batch effects by using approaches to achieve reliable and robust results. In this narrative review, we highlighted the advantages and disadvantages of both classes of DL approaches, and described key challenges to be addressed in future studies.

show abstract

“…Medical images are widely used for diagnosis and treatment planning. Manual qualitative evaluation by domain experts is the most common method for analyzing data from these images, which is time-consuming and prone to interobserver and intraobserver variabilities ( 1 , 2 ). Additionally, human interpretation may not fully leverage quantitative features unapparent to the naked eye.…”

Section: Introductionmentioning

confidence: 99%

Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls

Maleki¹,

Ovens²,

Gupta³

et al. 2023

Radiology: Artificial Intelligence

View full text Add to dashboard Cite

Purpose To investigate the impact of the following three methodological pitfalls on model generalizability: ( a ) violation of the independence assumption, ( b ) model evaluation with an inappropriate performance indicator or baseline for comparison, and ( c ) batch effect. Materials and Methods The authors used retrospective CT, histopathologic analysis, and radiography datasets to develop machine learning models with and without the three methodological pitfalls to quantitatively illustrate their effect on model performance and generalizability. F1 score was used to measure performance, and differences in performance between models developed with and without errors were assessed using the Wilcoxon rank sum test when applicable. Results Violation of the independence assumption by applying oversampling, feature selection, and data augmentation before splitting data into training, validation, and test sets seemingly improved model F1 scores by 71.2% for predicting local recurrence and 5.0% for predicting 3-year overall survival in head and neck cancer and by 46.0% for distinguishing histopathologic patterns in lung cancer. Randomly distributing data points for a patient across datasets superficially improved the F1 score by 21.8%. High model performance metrics did not indicate high-quality lung segmentation. In the presence of a batch effect, a model built for pneumonia detection had an F1 score of 98.7% but correctly classified only 3.86% of samples from a new dataset of healthy patients. Conclusion Machine learning models developed with these methodological pitfalls, which are undetectable during internal evaluation, produce inaccurate predictions; thus, understanding and avoiding these pitfalls is necessary for developing generalizable models. Keywords: Random Forest, Diagnosis, Prognosis, Convolutional Neural Network (CNN), Medical Image Analysis, Generalizability, Machine Learning, Deep Learning, Model Evaluation Supplemental material is available for this article. Published under a CC BY 4.0 license.

show abstract

Interobserver variability in quality assessment of magnetic resonance images

Cited by 17 publications

References 43 publications

Deep learning-based high-accuracy detection for lumbar and cervical degenerative disease on T2-weighted MR images

Deep learning-based high-accuracy detection for lumbar and cervical degenerative disease on T2-weighted MR images

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls

Contact Info

Product

Resources

About