Generalizability of Deep Learning Segmentation Algorithms for Automated Assessment of Cartilage Morphology and MRI Relaxometry

Abstract

Background: Deep learning (DL)-based automatic segmentation models can expedite manual segmentation yet require resource-intensive fine-tuning before deployment on new datasets. The generalizability of DL methods to new datasets without fine-tuning is not well characterized. Purpose: Evaluate the generalizability of DL-based models by deploying pretrained models on independent datasets varying by MR scanner, acquisition parameters, and subject population. Study Type: Retrospective based on prospectively acquired data. Population: Overall test dataset: 59 subjects (26 females); Study 1: 5 healthy subjects (zero females), Study 2: 8 healthy subjects (eight females), Study 3: 10 subjects with osteoarthritis (eight females), Study 4: 36 subjects with various knee pathology (10 females). Field Strength/Sequence: A 3-T, quantitative double-echo steady state (qDESS). Assessment: Four annotators manually segmented knee cartilage. Each reader segmented one of four qDESS datasets in the test dataset. Two DL models, one trained on qDESS data and another on Osteoarthritis Initiative (OAI)-DESS data, were assessed. Manual and automatic segmentations were compared by quantifying variations in segmentation accuracy, volume, and T2 relaxation times for superficial and deep cartilage. Statistical Tests: Dice similarity coefficient (DSC) for segmentation accuracy. Lin's concordance correlation coefficient (CCC), Wilcoxon rank-sum tests, root-mean-squared error-coefficient-of-variation to quantify manual vs. automatic T2 and volume variations. Bland-Altman plots for manual vs. automatic T2 agreement. A P value < 0.05 was considered statistically significant. Results: DSCs for the qDESS-trained model, 0.79-0.93, were higher than those for the OAI-DESS-trained model, 0.59-0.79. T2 and volume CCCs for the qDESS-trained model, 0.75-0.98 and 0.47-0.95, were higher than respective CCCs for the OAI-DESS-trained model, 0.35-0.90 and 0.13-0.84. Bland-Altman 95% limits of agreement for superficial and deep cartilage T2 were lower for the qDESS-trained model, AE2.4 msec and AE4.0 msec, than the OAI-DESS-trained model, AE4.4 msec and AE5.2 msec. Data Conclusion:The qDESS-trained model may generalize well to independent qDESS datasets regardless of MR scanner, acquisition parameters, and subject population.

MiFL: Multi-Input Neural Networks in Federated Learning

“…the ability of a model to deal with unseen data reliably 28 . Various efforts have been made in the research community in this respect [29][30][31][32][33] . The studies show how higher exposure to data can increase generalizability, hence the need for real-life federations.…”

Evaluation of the consistency of the MRI- based AI segmentation cartilage model using the natural tibial plateau cartilage

“…An examination of the current body of scholarly work indicates that most of these evaluation metrics are founded on either distance or volume overlap. Among the evaluation measures based on volume overlap, the Dice similarity coefficient (DSC) is the most frequently employed metric [ 27 – 31 ]. Consequently, irrespective of the dataset employed, the DSC serves as this study's primary criterion for comparison.…”

MERGE: A model for multi-input biomedical federated learning

“… 24 Model generalizability has been extensively studied in ML literature. 25 , 26 , 27 , 28 , 29 The studies show how higher exposure to data can increase generalizability, hence the need for real-life federations. 25 , 26 , 27 , 28 , 29 FL provides a viable solution by enabling the virtual pooling of different datasets while maintaining data privacy and allowing the model to learn features appearing in examples from different data owners.…”