Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics

He, Tong; Kong, Ru; Holmes, Avram J.; Nguyen, Minh; Sabuncu, Mert R.; Eickhoff, Simon B.; Bzdok, Danilo; Feng, Jiashi; Yeo, B. T. Thomas

doi:10.1016/j.neuroimage.2019.116276

Cited by 214 publications

(250 citation statements)

References 88 publications

Supporting

Mentioning

233

Contrasting

Unclassified

Order By: Relevance

“…Finally, a third family directly applies general-purpose machine learning on sensor space signals without explicitly considering the data generating mechanism. Following a common trend in other areas of neuroimaging research (Dadi et al, 2019;Schulz et al, 2019;He et al, 2019), linear prediction methods have turned out extraordinarily well-suited for this task, i.e. , logistic regression (Andersen et al, 2015), linear discriminant analysis (Wardle et al, 2016), linear support vector machines (King et al, 2013).…”

Section: State-of-the Art Approaches To Predict From M/eeg Observationsmentioning

confidence: 99%

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Sabbagh

Ablin

Varoquaux

et al. 2019

Preprint

View full text Add to dashboard Cite

Predicting biomedical outcomes from Magnetoencephalography and Electroencephalography (M/EEG) is central to applications like decoding, brain-computer-interfaces (BCI) or biomarker development and is facilitated by supervised machine learning. Yet most of the literature is concerned with within-subject classification. Here, we focus on predicting continuous outcomes from M/EEG signal power across subjects. Considering different generative mechanisms for M/EEG signals and the biomedical outcome, we propose statistically-consistent predictive models that avoid sourcereconstruction based on the covariance as representation. Our mathematical analysis and ground truth simulations demonstrated that consistent parameter estimation can be obtained with Source Power Comodulation (SPoC) supervised spatial filtering or by embedding with Riemannian geometry. Additional simulations revealed that Riemannian methods were more robust to model violations, in particular geometric distortions induced by individual anatomy. To estimate the relative contribution of brain dynamics and anatomy to prediction performance, we propose a novel model inspection procedure based on biophysical forward modeling. Applied to cross-subject prediction, the analysis revealed that the Riemannian model better exploited anatomical information while sensitivity to brain dynamics was similar across methods. We then probed the robustness of the models across different data cleaning options. Environmental denoising was globally important but Riemannian models were strikingly robust and continued performing even without preprocessing. Our results suggest each method has its niche: SPoC is practical for within-subject prediction while the Riemannian model may enable simple end-to-end learning.

show abstract

Section: State-of-the Art Approaches To Predict From M/eeg Observationsmentioning

confidence: 99%

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Sabbagh

Ablin

Varoquaux

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Previous work from the neuroimaging community (He, Kong, et al 2018) has made similar claims in stating that "deep neural networks [do not] outperform kernel regression for functional connectivity prediction of behavior". Yet, their experimental analysis setup may not be able to fully dismiss the critique of insufficient hyperparameter optimization in deep models.…”

Section: Deep Learning Did Not Universally Improve Prediction Performmentioning

confidence: 94%

“…Our finding of unexhausted linear modeling reserve may have considerable ramifications. This is because systematic evaluations of modern machine learning in brain-imaging (He, Kong, et al 2018;Vieira et al 2017;Plis et al 2014) as well as a large amount of studies applying complex nonlinear models in brain-imaging often have operated under the implicit assumption that linear effects are already sufficiently characterized with their prediction scaling as sample size increases. Typically, carefully characterizing linear effects provides a solid basis to compare against more complex nonlinear models.…”

Section: Present Sample Sizes Are Too Small To Even Fully Exploit Linmentioning

confidence: 99%

“…As a practical consequence, the scarcity of exploitable nonlinear structure in our brainimaging data suggests that linear models will continue to play a central role in analyzing brain scans like MRI measurements, at least over the next few years (He, Kong, et al 2018;Bzdok & Ioannidis 2019;Smith & Nichols 2018). The additional representational expressivity that modern nonlinear models provide comes at the cost of an increased risk of overfitting (Varoquaux et al 2017) and typically added challenges in interpretability of the estimated models Lundberg & Lee 2017;Chen et al 2018).…”

Section: Limited Success Of Exploiting Nonlinear Information In Currementioning

confidence: 99%

“…Second, deep learning approaches have repeatedly been found to perform worse or indistinguishably well compared to simpler baseline models when predicting demographic or behavioral data (Cole et al 2017;Jang et al 2017;He, Kong, et al 2018). For instance, Cole and colleagues (2017) showed that deep convolutional neural networks did not outperform Gaussian process models when predicting brain age from structural MRI data in ~2,000 healthy participants.…”

Section: Deep Learning Did Not Universally Improve Prediction Performmentioning

confidence: 99%

See 2 more Smart Citations

Deep learning for brains?: Different linear and nonlinear scaling in UK Biobank brain images vs. machine-learning datasets

Schulz

Yeo

Vogelstein

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, deep learning has unlocked unprecedented success in various domains, especially in image, text, and speech processing. These breakthroughs may hold promise for neuroscience and especially for brain-imaging investigators who start to analyze thousands of participants. However, deep learning is only beneficial if the data have nonlinear relationships and if they are exploitable at currently available sample sizes. We systematically profiled the performance of deep models, kernel models, and linear models as a function of sample size on UK Biobank brain images against established machine learning references. On MNIST and Zalando Fashion, prediction accuracy consistently improved when escalating from linear models to shallow-nonlinear models, and further improved when switching to deep-nonlinear models. The more observations were available for model training, the greater the performance gain we saw. In contrast, using structural or functional brain scans, simple linear models performed on par with more complex, highly parameterized models in age/sex prediction across increasing sample sizes. In fact, linear models kept improving as the sample size approached ~10,000 participants.Our results indicate that the increase in performance of linear models with additional data does not saturate at the limit of current feasibility. Yet, nonlinearities of common brain scans remain largely inaccessible to both kernel and deep learning methods at any examined scale. Exploratory Research Space (OPSF449), RWTH Aachen. Simulations were performed with computing resources granted by RWTH Aachen University under project number "rwth0238".

show abstract

Clinical Promise of Brain-Phenotype Modeling

Greene

Constable

2023

JAMA Psychiatry

View full text Add to dashboard Cite

ImportanceAssessing the link between whole-brain activity and individual differences in cognition and behavior has the potential to offer insights into psychiatric disorder etiology and change the practice of psychiatry, from diagnostic clarification to intervention. To this end, recent application of predictive modeling to link brain activity to phenotype has generated significant excitement, but clinical applications have largely not been realized. This Review explores explanations for the as yet limited practical utility of brain-phenotype modeling and proposes a path forward to fulfill this clinical potential.ObservationsClinical applications of brain-phenotype models are proposed and will require coordinated collaboration across the relatively siloed fields of psychometrics and computational neuroscience. Such interdisciplinary work will maximize the reliability and validity of modeled phenotypic measures, ensuring that resulting brain-based models are interpretable and useful. The models, in turn, may shed additional light on the neurobiological systems into which each phenotypic measure taps, permitting further phenotype refinement.Conclusions and RelevanceTogether, these observations reflect an opportunity: bridging the divide between phenotypic measure development and validation and measure end use for brain-phenotype modeling holds the promise that each may inform the other, yielding more precise and useful brain-phenotype models. Such models can in turn be used to reveal the macroscale neural bases of a given phenotype, advancing basic neuroscientific understanding and identifying circuits that can be targeted (eg, via closed-loop neurofeedback or brain stimulation) to slow, reverse, or even prevent functional impairment.

show abstract

Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics

Cited by 214 publications

References 88 publications

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

Deep learning for brains?: Different linear and nonlinear scaling in UK Biobank brain images vs. machine-learning datasets

Clinical Promise of Brain-Phenotype Modeling

Contact Info

Product

Resources

About