Transferring and Generalizing Deep-Learning-based Neural Encoding Models across Subjects

Abstract

Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the specific subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while deep residual neural network driven by image recognition was used to model the visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish subject-specific or populationwide predictive models of cortical representations of high-dimensional and hierarchical visual features.

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

“…Combining features across these layers ended up with a very high‐dimensional feature space. To reduce the dimension of the feature space, principal component analysis (PCA) was applied first to each layer and then to all layers, similar to our prior studies (Wen et al, 2017a, 2017b, 2017c). The principal components (PCs) were identified based on the feature representations of the training movie, and explained 90% variance.…”

“…The prediction accuracy was quantified as the temporal correlation ( r ) between the observed and predicted responses at each voxel. As in our previous studies (Wen et al, 2017a, 2017b, 2017c), the statistical significance of the prediction accuracy was evaluated voxel‐by‐voxel with a block‐permutation test (Adolf et al, ) corrected at the false discovery rate (FDR) .…”

“…The experimental data were from our previous studies (Wen et al, 2017a(Wen et al, , 2017bWen, Shi, Chen, & Liu, 2017c), according to a research protocol approved by the Institutional Review Board at Purdue University. Briefly, we acquired fMRI scans from three healthy subjects while they were watching natural videos.…”

“…Similar to our prior studies (Wen et al, 2017a(Wen et al, , 2017b(Wen et al, , 2017c, a pretrained CNN, also known as the VGG16 (Simonyan & Zisserman, 2014), was used to extract the hierarchical feature representations of every video frame as the outputs of artificial neurons (or units). This CNN contained 16 layers of units stacked in a feedforward network for processing the spatial information in the input.…”

Decoding and mapping task states of the human brain via deep learning

“…Transfer learning is widely used for training DNNs with limited medical data (Sharif Razavian, Azizpour, Sullivan, & Carlsson, ). It takes advantage of similar data within big datasets (Ciompi et al, ; Kermany et al, ; Wen, Shi, Chen, & Liu, ). Recent large fMRI projects, such as the Human Connectome Project (HCP; Van Essen, et al, ) and BioBank (Miller et al, ), allow us to access massive amounts of fMRI data.…”