Reproducibility of importance extraction methods in neural network based fMRI classification

Abstract

Recent advances in machine learning allow faster training, improved performance and increased interpretability of classification techniques. Consequently, their application in neuroscience is rapidly increasing. While classification approaches have proved useful in functional magnetic resonance imaging (fMRI) studies, there are concerns regarding extraction, reproducibility and visualization of brain regions that contribute most significantly to the classification. We addressed these issues using an fMRI classification scheme based on neural networks and compared a set of methods for extraction of category-related voxel importances in three simulated and two empirical datasets. The simulation data revealed that the proposed scheme successfully detects spatially distributed and overlapping activation patterns upon successful classification. Application of the proposed classification scheme to two previously published empirical fMRI datasets revealed robust importance maps that extensively overlap with univariate maps but also provide complementary information. Our results demonstrate increased statistical power of importance maps compared to univariate approaches for both detection of overlapping patterns and patterns with weak univariate information.

Where in the brain do internally generated and externally presented visual information interact?

“…The motivation to analyze our data using a neural network classifier, instead of univariate analyses, such as GLM, is twofold: First, classification accuracies directly quantify the discriminability across conditions. Second, previous work has shown that importance maps generated from neural network classifiers can reveal multivariate patterns and patterns with low univariate information (Gotsopoulos et al, 2018). Such information may not be available when performing univariate analysis, such as GLM, or when feeding the classifier with data preprocessed by a univariate method, such as GLM coefficients.…”

“…A group brain mask was determined by selecting voxels that were present (i.e., had non-zero standard deviation values) in all participants, resulting in a total of 203477 voxels. Classification of different trial types was performed with a linear (i.e., no hidden layers) artificial neural network classifier, as implemented in an in-house developed neural network toolbox that has been previously used to classify fMRI data (Gotsopoulos et al, 2018; available at https://github.com/gostopa1/DeepNNs). The classifier utilized softmax activation function in the output layer and cross-entropy loss function.…”

“…However, instead of extracting the raw weights of the classifier, whose magnitude and sign have no unambiguous interpretation, we used an importance extraction method based on layer-wise relevance propagation that decomposes the output of the classifier back to the inputs, thus providing a direct interpretation of each importance map (Bach et al, 2015;Montavon et al, 2017). Such importance extraction method for neural network classifier was recently shown to provide increased statistical power compared to univariate approaches for both detection of overlapping patterns as well as patterns with weak univariate information (Gotsopoulos et al, 2018).…”

Sparse Deep Neural Network for Encoding and Decoding the Structural Connectome

“…To overcome these issues, relevancy (or saliency) backpropagation methods have been proposed in the literature [26] . Since all back-propagation based approaches depend on gradient computation, they face saturation problems and may produce misleading results at the discontinuities of activation functions [27] . Several attempts have been made recently to determine relevancy (importance, contributions, or saliency) of features that are most discriminative for classification using DNN models [26] .…”