Analyzing Neuroimaging Data Through Recurrent Deep Learning Models

Abstract

The application of deep learning (DL) models to neuroimaging data poses several challenges, due to the high dimensionality, low sample size and complex temporo-spatial dependency structure of these datasets. Even further, DL models act as as black-box models, impeding insight into the association of cognitive state and brain activity. To approach these challenges, we introduce the DeepLight framework, which utilizes long short-term memory (LSTM) based DL models to analyze whole-brain functional Magnetic Resonance Imaging (fMRI) data. To decode a cognitive state (e.g., seeing the image of a house), DeepLight separates the fMRI volume into a sequence of axial brain slices, which is then sequentially processed by an LSTM. To maintain interpretability, DeepLight adapts the layer-wise relevance propagation (LRP) technique. Thereby, decomposing its decoding decision into the contributions of the single input voxels to this decision. Importantly, the decomposition is performed on the level of single fMRI volumes, enabling DeepLight to study the associations between cognitive state and brain activity on several levels of data granularity, from the level of the group down to the level of single time points. To demonstrate the versatility of DeepLight, we apply it to a large fMRI dataset of the Human Connectome Project. We show that DeepLight outperforms conventional approaches of uniand multivariate fMRI analysis in decoding the cognitive states and in identifying the physiologically appropriate brain regions associated with these states. We further demonstrate DeepLight's ability to study the fine-grained temporo-spatial variability of brain activity over sequences of single fMRI samples.

Towards Explainable Artificial Intelligence

“…Only recently, it became possible to apply more powerful models such as deep neural networks without sacrificing interpretability. These explainable non-linear models have already attracted attention in domains such as neuroscience [87,89,20], health [33,14,40], autonomous driving [31], drug design [70] and physics [78,72] and it can be expected that they will play a pivotal role in future scientific research.…”

“…They not only outperform humans in complex visual tasks [16,53] or strategic games [56,83,61], but also became an indispensable part of our every day lives, e.g., as intelligent cell phone cameras which can recognize and track faces [71], as online services which can analyze and translate written texts [11] or as consumer devices which can understand speech and generate human-like answers [90]. Moreover, machine learning and artificial intelligence have become indispensable tools in the sciences for tasks such as prediction, simulation or exploration [78,15,89,92]. These immense successes of AI systems mainly became possible through improvements in deep learning methodology [48,47], the availability of large databases [17,34] and computational gains obtained with powerful GPU cards [52].…”

Deep Transfer Learning for Whole-Brain FMRI Analyses

“…1. Illustration of the DeepLight framework [13]. DeepLight first separates a wholebrain fMRI volume into a sequence of axial slices.…”

“…Training All DeepLight variants that were used in this study were trained as follows (if not reported otherwise): We iteratively trained DeepLight through backpropagation, by the use of the ADAM optimization algorithm, as implemented in tensorflow 1.13. During parameter estimation, we applied dropout regularization to all network layers as follows: We set the dropout probability to 50 % for the LSTM unit and softmax output layer, For the convolution layers, however, we set the dropout probability to 0% for the first four convolution layers, 20% for the next four convolution layers, and 40% for the last four convolution layers (in line with [13]). We further used a learning rate of 1e −4 and a batch size of 24 fMRI volumes.…”

“…In particular, we show that a DL model that has been trained on the data of six out of seven task-fMRI datasets of the Human Connectome Project database [1] performs better in decoding the cognitive states underlying a seventh, unrelated task, when compared to a model variant that is trained entirely from scratch on the data of this task. For this comparison, we utilize the DeepLight framework [13], which decodes a cognitive state from whole-brain fMRI data, by combining convolutional and recurrent DL elements (see Fig. 1 and Section 2.2).…”