Evaluating deep transfer learning for whole-brain cognitive decoding

Abstract

Research in many fields has shown that transfer learning (TL) is well-suited to improve the performance of deep learning (DL) models in datasets with small numbers of samples. This empirical success has triggered interest in the application of TL to cognitive decoding analyses with functional neuroimaging data. Here, we systematically evaluate TL for the application of DL models to the decoding of cognitive states (e.g., viewing images of faces or houses) from whole-brain functional Magnetic Resonance Imaging (fMRI) data. We first pre-train two DL architectures on a large, public fMRI dataset and subsequently evaluate their performance in an independent experimental task and a fully independent dataset. The pre-trained models consistently achieve higher decoding accuracies and generally require less training time and data than model variants that were not pre-trained, clearly underlining the benefits of pretraining. We demonstrate that these benefits arise from the ability of the pre-trained models to reuse many of their learned features when training with new data, providing deeper insights into the mechanisms giving rise to the benefits of pre-training. Yet, we also surface nuanced challenges for whole-brain cognitive decoding with DL models when interpreting the decoding decisions of the pre-trained models, as these have learned to utilize the fMRI data in unforeseen and counterintuitive ways to identify individual cognitive states.

Comparing interpretation methods in mental state decoding analyses with deep learning models

“…Note that we chose mutual information as a similarity measure, because the association between the group-level BOLD and attribution maps does not need to be linear. For example, it is possible that a DL model learns to identify a mental state through the activity of voxels that are meaningfully more active in this state as well as the activity of voxels that are meaningfully less active, resulting in an attribution map that assigns high relevance to voxels that exhibit high positive and negative values in a GLM analysis of the BOLD data (see Thomas et al, 2021a).…”

“…Accordingly, their learned mappings between input data and decoding decisions can be highly-complex and counterintuitive. For example, recent empirical work has shown that DL methods trained in mental state decoding analyses can identify individual mental states through voxels that exhibit meaningfully stronger activity in these states as well as voxels with meaningfully reduced activity in these states, leading to explanations that assign high attribution scores to voxels that receive both positive and negative weights in a standard GLM contrast analysis of the same BOLD data (Thomas et al, 2021a). It is thus essential to always interpret any explanations of an interpretation method in the light of the results of standard analyses of the same BOLD data (e.g., with linear models Friston et al, 1994, Grosenick et al, 2013, Kriegeskorte et al, 2006 as well as related empirical findings (e.g., as provided by NeuroSynth; Yarkoni et al, 2011), to understand how a model's weighting of the input in its decoding decision relates to the characteristics of the input data.…”

Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data

“…A wealth of empirical evidence has demonstrated that models pre-trained on large neuroimaging datasets generally achieve better mental state decoding performance on new data, while requiring less training time and data, when compared to models trained from scratch [e.g., 6,7,8]. Yet, much of this work has either pre-trained models on large but homogenous datasets, such as data from many individuals who all perform the same few tasks at the same few acquisition sites [e.g., by using data from the Human Connectome Project; 7,9] or trained models on highly-preprocessed input data [e.g., statistical maps summarizing the measured sequences of brain activity; 10], both limiting the downstream generalizability of the pre-trained models.…”

On the benefits of self-taught learning for brain decoding

“…Many studies were also made on inductive transfer learning with labeled source data as defined in [ 19 ] (e.g., source task and target task are different, as well as source domain and target domain) [ 28–30 ]. For instance, Thomas & al.…”

“…For instance, Thomas & al. [ 28 ] pretrained 2 DL classifiers on a large, public whole-brain fMRI dataset of the HCP, fine-tuned them, and evaluated their performance on another task on the same dataset and on a fully independent dataset. In another study, Gao & al.…”