Deep image reconstruction from human brain activity

Abstract

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

How are visual words represented? Insights from EEG‐based visual word decoding, feature derivation and image reconstruction

“…Second, visual word features were derived directly from the structure of the EEG data and used for the purpose of word image reconstruction. Previous work has reconstructed single characters such as letters from fMRI patterns in visual cortex (Schoenmakers et al, 2013;Shen et al, 2019;Thirion et al, 2006) or visualized their representation through psychophysical methods (Gosselin & Schyns, 2003)-see also complementary work (Pasley et al, 2012) targeting ECoG-based speech reconstruction. In contrast, the current results demonstrate, for the first time to our knowledge, the ability to reconstruct the visual appearance of whole words from neural recordings.…”

“…Relevantly here, neural-based image reconstruction (Chang & Tsao, 2017;Naselaris, Prenger, Kay, Oliver, & Gallant, 2009;Nestor, Plaut, & Behrmann, 2016;Nishimoto et al, 2011;Shen, Horikawa, Majima, & Kamitani, 2019) aims to reveal the content of fine-grained visual representations by retrieving the appearance of visual objects from neural activity prompted by their processing. For instance, several fMRI studies have addressed the challenge of reconstructing the appearance of single letters from fMRI patterns associated with their reading (Miyawaki et al, 2008;Schoenmakers, Barth, Heskes, & van Gerven, 2013;Thirion et al, 2006).…”

Characterization of deep neural network features by decodability from human brain activity

“…The fMRI data made it possible to decode DNN feature values from the brain activity patterns, and the estimated decodability was highly consistent across subjects. Thus, the present dataset could provide an opportunity to utilized for various purposes, including the feature selection in neural encoding and decoding analyses 4,8,10 and further applications by combining the decoded features with deep neural network technology 6,9 .…”

“…On the basis of the hierarchical representational similarity between the brain and DNNs, our recent study demonstrated that human brain activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into DNN feature values 6 . Combining those decoded DNN features and techniques developed with DNNs, recent work has started to develop new technologies to read out richer contents in the brain as demonstrated in the generic decoding of seen, imagined, and dreamed objects 6,7 and in the reconstruction of seen and imagined images 9 . As exemplified by these studies, decoding of DNN features from brain activity patterns can then provide opportunities to develop new technologies for further applications in brain machine interfacing.…”

Transfer learning of deep neural network representations for fMRI decoding

“…In [15] authors presented an encoding model by which, starting by Convolutional Neural Network (CNN) layer activations and using ridge regression with linear kernel, they predict BOLD fMRI response, employing two different databases ([11] and [16]). In [17] the authors presented a novel image reconstruction method, in which the pixel values of an image are optimised to make its CNN features similar to those decoded from human brain activity at multiple layers. A further example of encoding came from [18], in which the prediction of brain response is done multi-subject and using Bayesian incremental learning.Whereas encoding models have greatly benefited from the inclusion of DNN-derived features in the modeling pipeline, decoding models have not yet exploited the full potential offered by them.…”