Promises, Pitfalls, and Basic Guidelines for Applying Machine Learning Classifiers to Psychiatric Imaging Data, with Autism as an Example

Abstract

Most psychiatric disorders are associated with subtle alterations in brain function and are subject to large interindividual differences. Typically, the diagnosis of these disorders requires time-consuming behavioral assessments administered by a multidisciplinary team with extensive experience. While the application of Machine Learning classification methods (ML classifiers) to neuroimaging data has the potential to speed and simplify diagnosis of psychiatric disorders, the methods, assumptions, and analytical steps are currently opaque and not accessible to researchers and clinicians outside the field. In this paper, we describe potential classification pipelines for autism spectrum disorder, as an example of a psychiatric disorder. The analyses are based on resting-state fMRI data derived from a multisite data repository (ABIDE). We compare several popular ML classifiers such as support vector machines, neural networks, and regression approaches, among others. In a tutorial style, written to be equally accessible for researchers and clinicians, we explain the rationale of each classification approach, clarify the underlying assumptions, and discuss possible pitfalls and challenges. We also provide the data as well as the MATLAB code we used to achieve our results. We show that out-of-the-box ML classifiers can yield classification accuracies of about 60–70%. Finally, we discuss how classification accuracy can be further improved, and we mention methodological developments that are needed to pave the way for the use of ML classifiers in clinical practice.

Neurobiological support to the diagnosis of ADHD in stimulant‐naïve adults: pattern recognition analyses of MRI data

“…Our pattern recognition algorithms was trained and tested on groups of a relatively modest size and, therefore, accuracies that are derived from those samples are not directly representative for predictions in clinical populations . The performance indices of fair accuracy obtained in the present study are consistent with the notion that there are important limitations in the application of ML techniques for predicting diagnosis in clinical neuroimaging research . Despite several promising results , there are many challenges that still need to be addressed before these techniques can be seen as promptly applicable to make psychiatric diagnostic predictions.…”

A systematic review of structural MRI biomarkers in autism spectrum disorder: A machine learning perspective

“…Consequently, there have been a wide range of structural brain regions implicated with ASD, most commonly that of early brain overgrowth and head circumference (Mosconi et al, 2009; Sacco et al, 2015), as well as more localised brain regions that may be associated with the social and motor impairments characteristic of ASD, including the frontal lobes, amygdala, cerebellum (Amaral et al, 2008; Li et al, 2017; Sivapalan and Aitchison, 2014), corpus callosum (Bellani et al, 2013; Hrdlicka, 2008; Stigler et al, 2011) and basal ganglia (Calderoni et al, 2014; Dougherty et al, 2016a). However, there has not yet been an agreement on structural changes in the brain that reflect these underlying mechanisms of ASD, limiting the utility of machine learning to provide accurate diagnoses of ASD and patient prognoses (Kassraian‐Fard et al, 2016).…”

Is there any incremental benefit to conducting neuroimaging and neurocognitive assessments in the diagnosis of ADHD in young children? A machine learning investigation

“…While computationally more challenging, this latter approach enables researchers to answer key questions regarding mental representations (e.g., what type of information is represented in different brain regions and at different stages of cognitive processing) in cognitive neuroscience research (Haxby et al, 2001;Kuhl et al, 2011;Liang et al, 2013;Norman et al, 2006;Oztekin & Badre, 2011;Polyn et al, 2005)], and provides unique opportunities for clinical diagnosis and prediction of disorder or symptom severity, as well as behavioral outcomes associated with specific disorders. Despite this promising potential for such computational, multivariate approaches to advance translational neuroscience, significant challenges and pitfalls have prevented the development of generalizable methods and approaches that can be applied in clinical settings [see (Arbabshirani et al, 2017;Kassraian-Fard et al, 2016;Varoquaux, 2018;Varoquaux et al, 2017;Woo et al, 2017)].…”

“…Accordingly, overfitting is a major obstacle in generating a valid predictive model that can produce generalizable results for clinical diagnosis, and which can be used across a variety of clinical settings (Arbabshirani et al, 2017;Foster et al, 2014;Kassraian-Fard et al, 2016;Pulini et al, 2019;Varoquaux, 2018;Varoquaux et al, 2017;Woo et al, 2017). Indeed, most neuroscience studies suffer from small sample sizes (Arbabshirani et al, 2017;Foster et al, 2014;Kassraian-Fard et al, 2016;Pulini et al, 2019;Varoquaux, 2018). Notably, the current investigation leveraged a data set with a larger sample size (162 participants) that provided a unique opportunity to leverage predictive modeling that utilized a machine learning approach.…”