Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks

Abstract

Feedforward deep neural networks (DNN), artificial neural networks with multiple hidden layers, have recently demonstrated a record-breaking performance in multiple areas of applications in computer vision and speech processing. Following the success, DNNs have been applied to neuroimaging modalities including functional/structural magnetic resonance imaging (MRI) and positron-emission tomography data. However, no study has explicitly applied DNNs to 3D whole-brain fMRI volumes and thereby extracted hidden volumetric representations of fMRI that are discriminative for a task performed as the fMRI volume was acquired. Our study applied fully connected feedforward DNN to fMRI volumes collected in four sensorimotor tasks (i.e., left-hand clenching, right-hand clenching, auditory attention, and visual stimulus) undertaken by 12 healthy participants. Using a leave-one-subject-out cross-validation scheme, a restricted Boltzmann machine-based deep belief network was pretrained and used to initialize weights of the DNN. The pretrained DNN was fine-tuned while systematically controlling weight-sparsity levels across hidden layers. Optimal weight-sparsity levels were determined from a minimum validation error rate of fMRI volume classification. Minimum error rates (mean ± standard deviation; %) of 6.9 (± 3.8) were obtained from the three-layer DNN with the sparsest condition of weights across the three hidden layers. These error rates were even lower than the error rates from the single-layer network (9.4 ± 4.6) and the two-layer network (7.4 ± 4.1). The estimated DNN weights showed spatial patterns that are remarkably task-specific, particularly in the higher layers. The output values of the third hidden layer represented distinct patterns/codes of the 3D whole-brain fMRI volume and encoded the information of the tasks as evaluated from representational similarity analysis. Our reported findings show the ability of the DNN to classify a single fMRI volume based on the extraction of hidden representations of fMRI volumes associated with tasks across multiple hidden layers. Our study may be beneficial to the automatic classification/diagnosis of neuropsychiatric and neurological diseases and prediction of disease severity and recovery in (pre-) clinical settings using fMRI volumes without requiring an estimation of activation patterns or ad hoc statistical evaluation.

Decoding and mapping task states of the human brain via deep learning

“…These important features allow researchers to efficiently model complex systems without the burden of model/prior knowledge selection, especially in cases where too many features exist, as when analyzing medical images (Shen et al, ). Thus, DNNs are widely used by researchers for medical image analysis, such as brain image segmentation (Havaei et al, ; Wachinger et al, ; Zhang et al, ), neurology and psychiatric diagnostics (Hosseini‐Asl, Keynton, & El‐Baz, ; Meszlenyi, Buza, & Vidnyanszky, ; Plis et al, ; Vieira et al, ), brain state decoding (Jang et al, ), and brain computer interfaces (Schirrmeister et al, ).…”

“…A variety of deep methods have been applied to fMRI data, such as the autoencoder (Kim, Calhoun, Shim, & Lee, 2016), deep belief network (DBN; Jang et al, 2017;Plis et al, 2014), long short-term memory (LSTM) recurrent neural network (RNN; Li & Fan, 2019), and 2D CNN (Meszlenyi et al, 2017). Although the autoencoder is known to be efficient, especially when the dataset is small, it over- (c) Accuracy of fivefold cross-validation classification on the motor task on a small dataset.…”

“…For years, researchers have been attempting to decode and identify functions of the human brain based on functional brain imaging data (Dehaene et al, 1998;Haynes & Rees, 2006;Jang, Plis, Calhoun, & Lee, 2017;Poldrack, Halchenko, & Hanson, 2009;Rubin et al, 2017). The most popular among these brain-decoding methods is the support vector machine (SVM)-based multi-voxel pattern analysis (MVPA), a supervised technology that incorporates information from multiple variables at the same time (Kim & Oertzen, 2018;Kriegeskorte & Bandettini, 2007;Kriegeskorte, Goebel, & Bandettini, 2006;Norman, Polyn, Detre, & Haxby, 2006).…”

Deep learning for brains?: Different linear and nonlinear scaling in UK Biobank brain images vs. machine-learning datasets

“…Second, deep learning approaches have repeatedly been found to perform worse or indistinguishably well compared to simpler baseline models when predicting demographic or behavioral data (Cole et al 2017;Jang et al 2017;He, Kong, et al 2018). For instance, Cole and colleagues (2017) showed that deep convolutional neural networks did not outperform Gaussian process models when predicting brain age from structural MRI data in ~2,000 healthy participants.…”

Brain State Decoding Based on fMRI Using Semisupervised Sparse Representation Classifications

“…If sufficient training samples are available from each class, SRC will be possible to represent the test samples as a linear combination of the training samples from the same class. Although various supervised-learning classification techniques that included support vector machine (SVM), logistic regression, naïve Bayesian, and deep neural networks were applied to brain state decoding of fMRI data [6–9], SRC has seldom been applied to fMRI-based brain state decoding due to the various variabilities in fMRI data, such as complex and high noises and the delay of hemodynamic response. Given the promising outcomes of SRC in other research fields, it is necessary to explore the effective use of SRC in fMRI analysis.…”