Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity

Abstract

Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.

Towards Naturalistic Speech Decoding from Intracranial Brain Data

“…More robust brain decoding that can tackle continuous speech in noisy realistic environments is needed. In the field of vision, most recent work has shown promise of GANs in photo-realistic reconstruction of images from the brain activity [12]- [14]. Instead of decoding complex highdimensional data such as natural images directly from the brain, pretrained GANs were used to generate this output from a compressed latent vector of only 100 − 300 values.…”

Generative adversarial networks unlock new methods for cognitive science

“…Furthermore, the related problem of text to image translation has also been tackled with GANs [31][32][33], although the most astonishing results on that task are from the Dall-E model [10]. Moreover, GANs have been used to translate functional MRI (fMRI) data back to the presented visual stimulus that evoked it [34][35][36][37][38][39][40], which is useful for uncovering internal neural representations.…”

High-resolution image reconstruction with latent diffusion models from human brain activity

“…Reconstructing visual images from brain activity, such as that measured by functional Magnetic Resonance Imaging (fMRI), is an intriguing but challenging problem, because the underlying representations in the brain are largely unknown, and the sample size typically associated with brain data is relatively small [17, 26, 30, 32]. In recent years, researchers have started addressing this task using deep-learning models and algorithms, including generative adversarial networks (GANs) and self-supervised learning [2, 5, 7, 24, 25, 27, 36,44–46]. Additionally, more recent studies have increased semantic fidelity by explicitly using the semantic content of images as auxiliary inputs for reconstruction [5,25].…”

“…Recent developments in the measurement of population brain activity, combined with advances in the implementation and design of deep neural network models, have allowed direct comparisons between latent representations in biological brains and architectural characteristics of artificial networks, providing important insights into how these systems operate [3, 8–10, 13, 18, 19, 21, 42, 43, 54, 55]. These efforts have included the reconstruction of visual experiences (perception or imagery) from brain activity, and the examination of potential correspondences between the computational processes associated with biological and artificial systems [2,5,7,24,25,27,36,44–46].…”