Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Abstract

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

“…The use of CNNs is ubiquitous in image processing tasks, including image reconstruction. Specifically, encoder-decoder (Beliy et al, 2019;Gaziv et al, 2020), U-Net (Fang et al, 2020), generative adversarial network (Goodfellow et al, 2014), and variational autoencoder (Kingma and Welling, 2014) are popular architectures that adopt stacked convolutional layers to extract features at multiple levels. Shen et al (2019b) utilized a pretrained VGG-19-based DNN to extract hierarchical features from stimuli images (see Figure 3A).…”

“…In a follow-up study, Gaziv et al (2020) improved the reconstruction accuracy of BeliyEncDec by introducing a loss function based on the perceptual similarity measure (Zhang et al, 2018). To calculate perceptual similarity loss, the authors first extracted multilayer features from original and reconstructed images using VGG and then compared the extracted features layerwise.…”

“…To calculate perceptual similarity loss, the authors first extracted multilayer features from original and reconstructed images using VGG and then compared the extracted features layerwise. To distinguish it from BeliyEncDec, we refer to the framework proposed by Gaziv et al (2020) as GazivEncDec.…”

“…candidate images, subjects are instructed to choose the one that appears to have a higher resemblance to the original. Such behavioral studies can be conducted by employing human evaluators or using Amazon Mechanical Turk 6 (Seeliger et al, 2018;Gaziv et al, 2020).…”

“…Despite the wide adoption of SSIM as a perceptual metric, it compares poorly with many characteristics of human perception (Zhang et al, 2018). Several studies, including Güçlütürk et al (2017), Qiao et al (2018), Mozafari et al (2020), andGaziv et al (2020), emphasize the importance of higher-level perceptual similarity over lower-level metrics in evaluation because of the better correspondence of higher-level perceptual similarity to human perceptual judgments (Zhang et al, 2018).…”