Insights from an autism imaging biomarker challenge: promises and threats to biomarker discovery

Abstract

MRI has been extensively used to identify anatomical and functional differences in Autism Spectrum Disorder (ASD). Yet, many of these findings have proven difficult to replicate because studies rely on small cohorts and are built on many complex, undisclosed, analytic choices. We conducted an international challenge to predict ASD diagnosis from MRI data, where we provided preprocessed anatomical and functional MRI data from > 2,000 individuals. Evaluation of the predictions was rigorously blinded. 146 challengers submitted prediction algorithms, which were evaluated at the end of the challenge using unseen data and an additional acquisition site. On the best algorithms, we studied the importance of MRI modalities, brain regions, and sample size. We found evidence that MRI could predict ASD diagnosis: the 10 best algorithms reliably predicted diagnosis with AUC∼0.80 – far superior to what can be currently obtained using genotyping data in cohorts 20-times larger. We observed that functional MRI was more important for prediction than anatomical MRI, and that increasing sample size steadily increased prediction accuracy, providing an efficient strategy to improve biomarkers. We also observed that despite a strong incentive to generalise to unseen data, model development on a given dataset faces the risk of overfitting: performing well in cross-validation on the data at hand, but not generalising. Finally, we were able to predict ASD diagnosis on an external sample added after the end of the challenge (EU-AIMS), although with a lower prediction accuracy (AUC=0.72). This indicates that despite being based on a large multisite cohort, our challenge still produced biomarkers fragile in the face of dataset shifts.HighlightsMRI can provide reliable biomarkers for ASDBecause of overfitting, models with excellent apparent performance on public data failed to generaliseThe 10 best algorithms used classic machine learning methodsBiomarker development needs blinded evaluation.Prediction performance should reach AUC=0.83 on samples with 10k subjects

Towards 3D Deep Learning for neuropsychiatry: predicting Autism diagnosis using an interpretable Deep Learning pipeline applied to minimally processed structural MRI data

“…Our findings show that these algorithms are capable of inferring Autism diagnosis on the basis of structural MRIs with at least the same level of accuracy as traditional Machine Learning algorithms, while requiring a smaller number of training epochs. The average accuracy score (64.1%) and ROC AUC score (0.67) obtained for participants without comorbidities is consistent with previous Machine Learning models trained on sMRI data (e.g., [39]). The comparable accuracy we achieved should be viewed in the context of the speed of inference of Deep Learning models over Machine Learning approaches.…”

“…There is considerable scope to extend our interpretable Deep Learning pipeline to the prediction of other neurological or neuropsychiatric conditions or to other MRI modalities. Traut et al[39] reported that prediction of Autism was considerably improved (from AUC=0.66 using only anatomical MRI to AUC=0.79 using both anatomical and functional data) for a blended model that incorporated both functional and structural MRI data. Future work will examine whether functional MRI data can also improve our models.…”

“…Traut et al [39] reported that prediction of Autism was considerably improved (from AUC=0.66 using only anatomical MRI to AUC=0.79 using both anatomical and functional data) for a blended model that incorporated both functional and structural MRI data. Future work will examine whether functional MRI data can also improve our models.…”

“…18.22281196 doi: medRxiv preprint approaches have had moderate success, with recently reported prediction accuracies (for Autism diagnosis) in the range of 65-70% for models built using both functional and structural MRI data [35][36][37][38]. In an effort to boost accuracy through competition, Traut et al [39] held an international challenge in which competing teams predicted Autism diagnosis using a large multisite dataset comprising preprocessed anatomical and functional MRI data from > 2,000 individuals. Of the 589 models submitted, the 10 best were combined and evaluated using a subset of unseen data (from one of the sites included in the main dataset), as well as data from an additional, independent acquisition site.…”

Generalizable neuromarker for autism spectrum disorder across imaging sites and developmental stages: A multi-site study

“…; https://doi.org/10.1101/2023.03. 26.534053 doi: bioRxiv preprint here we call neuromarkers, hold promise in their potential to achieve greater classification accuracy with fewer participants than genetic biomarkers, which only explain 2.45% of risk variance even with more than 10,000 cases (9,10). However, there are several challenges to the development of robust neuromarkers.…”