Critical Appraisal of Artificial Intelligence–Enabled Imaging Tools Using the Levels of Evidence System

Pham, Ngoc Thi; Hill, Virginia; Rauschecker, Andreas M.; Lui, Yvonne W.; Niogi, Sumit N.; Filippi, C.G.; Chang, Peter D.; Zaharchuk, Greg; Wintermark, Max

doi:10.3174/ajnr.a7850

Cited by 9 publications

(3 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In accordance with the previously reported critical appraisal guidelines for AI research, our study aligns with Level 5B: one retrospective study with only internal data used for final performance reporting. [28] Our understanding of surgical techniques and adjuvant therapy has gradually evolved. Patients treated in the latter years of this study likely benefited from our greater knowledge and improved treatments that are not included as clinical parameters in this study.…”

Section: Discussionmentioning

confidence: 99%

Postoperative Karnofsky performance status prediction in patients withIDHwild-type glioblastoma: a multimodal approach integrating clinical and deep imaging features

Sasagasako,

Ueda,

Mineharu

et al. 2024

Preprint

View full text Add to dashboard Cite

Background and Purpose: Glioblastoma is a highly aggressive brain tumor with limited survival that poses challenges in predicting patient outcomes. The Karnofsky Performance Status (KPS) score is a valuable tool for assessing patient functionality and contributes to the stratification of patients with poor prognoses. This study aimed to develop a 6-month postoperative Karnofsky Performance Status (KPS) prediction model by combining clinical data with deep learning-based image features from pre- and postoperative MRI scans, offering enhanced personalized care for glioblastoma patients. Materials and Methods: Using 1,476 MRI datasets from the Brain Tumor Segmentation Challenge 2020 public database, we pretrained two variational autoencoders (VAEs). Imaging features from the latent spaces of the VAEs were used for KPS prediction. Neural network-based KPS prediction models were developed to predict scores below 70 at 6 months postoperatively. In this retrospective single-center analysis, we incorporated clinical parameters and pre- and postoperative MRI images from 150 newly diagnosed IDH wild-type glioblastoma, divided into training (100 patients) and test (50 patients) sets. In training set, the performance of these models was evaluated using the area under the curve (AUC), calculated through fivefold cross-validation repeated 10 times. The performance of these models in the training set was evaluated using the area under the curve (AUC) from a fivefold cross-validation repeated 10 times. The final evaluation of the developed models assessed in the test set. Results: Among the 150 patients, 61 had 6-month postoperative KPS scores below 70 and 89 scored 70 or higher. We developed three models: a clinical-based model, an MRI-based model, and a multimodal model that incorporated both clinical parameters and MRI features. In the training set, the mean AUC was 0.785±0.051 for the multimodal model, which was significantly higher than the clinical-based model (0.716±0.059, P=0.038) using only clinical parameters and MRI-based model (0.651±0.028, P<0.001) using only MRI features. In the test set, the multimodal model achieved an AUC of 0.810, outperforming the clinical-based (0.670) and MRI-based (0.650) models. Conclusion: The integration of MRI features extracted from VAEs with clinical parameters in the multimodal model substantially enhanced KPS prediction performance. This approach has the potential to improve prognostic prediction, paving the way for more personalized and effective treatments for patients with glioblastoma. Abbreviations: KPS, Karnofsky performance status. IDH, isocitrate dehydrogenase. VAE, variational autoencoder. BraTS, Brain Tumor Segmentation challenge

show abstract

Section: Discussionmentioning

confidence: 99%

Postoperative Karnofsky performance status prediction in patients withIDHwild-type glioblastoma: a multimodal approach integrating clinical and deep imaging features

Sasagasako,

Ueda,

Mineharu

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…This research constitutes Level 5A evidence (data quality and AI model development with external testing) because it represents 1 retrospective study with internal and external data used for final performance reporting. 29…”

Section: Methodsmentioning

confidence: 99%

A Radiomic “Warning Sign” of Progression on Brain MRI in Individuals with MS

Kelly,

Mathur,

McGuinness

et al. 2024

AJNR Am J Neuroradiol

View full text Add to dashboard Cite

BACKGROUND AND PURPOSE: MS is a chronic progressive, idiopathic, demyelinating disorder whose diagnosis is contingent on the interpretation of MR imaging. New MR imaging lesions are an early biomarker of disease progression. We aimed to evaluate a machine learning model based on radiomics features in predicting progression on MR imaging of the brain in individuals with MS. MATERIALS AND METHODS:This retrospective cohort study with external validation on open-access data obtained full ethics approval. Longitudinal MR imaging data for patients with MS were collected and processed for machine learning. Radiomics features were extracted at the future location of a new lesion in the patients' prior MR imaging ("prelesion"). Additionally, "control" samples were obtained from the normal-appearing white matter for each participant. Machine learning models for binary classification were trained and tested and then evaluated the external data of the model. RESULTS:The total number of participants was 167. Of the 147 in the training/test set, 102 were women and 45 were men. The average age was 42 (range, 21-74 years). The best-performing radiomics-based model was XGBoost, with accuracy, precision, recall, and F1-score of 0.91, 0.91, 0.91, and 0.91 on the test set, and 0.74, 0.74, 0.74, and 0.70 on the external validation set. The 5 most important radiomics features to the XGBoost model were associated with the overall heterogeneity and low gray-level emphasis of the segmented regions. Probability maps were produced to illustrate potential future clinical applications. CONCLUSIONS:Our machine learning model based on radiomics features successfully differentiated prelesions from normalappearing white matter. This outcome suggests that radiomics features from normal-appearing white matter could serve as an imaging biomarker for progression of MS on MR imaging.

show abstract

“…We received full IRB approval and the requirement for prospective consent was waived. This research constitutes Level 5A evidence (Data quality and AI model development with external testing) as it represents one retrospective study with internal and external data used for final reporting 31 . Subsections of this cohort have been published previously but under different experimental conditions with different research questions.…”

Section: Study Design and Data Sourcesmentioning

confidence: 99%

A Siamese U-Transformer for change detection on MRI brain for multiple sclerosis, a model development and external validation study

Kelly,

Mathur,

Killeen

et al. 2024

Preprint

View full text Add to dashboard Cite

Background Multiple Sclerosis (MS), is a chronic idiopathic demyelinating disorder of the CNS. Imaging plays a central role in diagnosis and monitoring. Monitoring for progression however, can be repetitive for neuroradiologists, and this has led to interest in automated lesion detection. Simultaneously, in the computer science field of Remote Sensing, Change Detection (CD), the identification of change between co-registered images at different times, has been disrupted by the emergence of Vision Transformers. CD offers an alternative to semantic segmentation leveraging the temporal information in the data. Methods In this retrospective study with external validation we reframe the clinical radiology task of new lesion identification as a CD problem. Consecutive patients who had MRI studies for MS at our institution between 2019 and 2022 were reviewed and those with new lesion(s) were included. External data was obtained from the MSSEG2 challenge and OpenMS. Multiple CD models, and a novel model (NeUFormer), were trained and tested. Results were analysed on both paired slices and at the patient level. Expected Cost (EC) and F2 were independently and prospectively chosen as our primary evaluation metrics. For external data we report DICE and F1 to allow for comparison with existing data. For each test set 1000 bootstrapping simulations were performed by sampling 10 patient samples with replacement giving a non parametric estimate of the confidence interval. Wilcoxon statistics were calculated to test for significance. Findings 43,440 MR images were included for analysis (21,720 pairs). The internal set comprised of 170 patients (110 for training, 30 for tuning, 30 testing) with 120 females and 50 males, average age of 42 (range 21,74). 60 (40 + 20) patients were included for external validation. In the CD experiments (2D) our proposed NeuFormer model achieved the best (lowest) Expected Cost (EC) (p=0.0095), the best F2 and second best DICE (p<0.0001). At the patient level our NeUFormer model had the joint highest number of True Positive lesions, and lowest number of False negatives (p<0.002). For CD on external data, NeUFormer achieved the highest DICE on both datasets (p<0.0001). NeUFormer had the lowest or joint lowest number of False Positives on external data (p<0.0001 in all cases). Interpretation Reformulating new lesion identification as a CD problem allows the use of new techniques and methods of evaluation. We introduce a novel Siamese U-Transformer, NeUFormer, which combines concepts from U-Net, Siamese Networks, and vision transformers to create a model with improved small lesion detection and the consistently best EC. Its ability to increase detection of small lesions, balanced with relatively few false positives, and superior generalisability has the potential to greatly impact the field of the identification of radiologic progression of MS with AI.

show abstract

Critical Appraisal of Artificial Intelligence–Enabled Imaging Tools Using the Levels of Evidence System

Cited by 9 publications

References 55 publications

Postoperative Karnofsky performance status prediction in patients withIDHwild-type glioblastoma: a multimodal approach integrating clinical and deep imaging features

Postoperative Karnofsky performance status prediction in patients withIDHwild-type glioblastoma: a multimodal approach integrating clinical and deep imaging features

A Radiomic “Warning Sign” of Progression on Brain MRI in Individuals with MS

A Siamese U-Transformer for change detection on MRI brain for multiple sclerosis, a model development and external validation study

Contact Info

Product

Resources

About