End-to-End Deep Image Reconstruction From Human Brain Activity

Shen, Guohua; Dwivedi, Kshitij; Majima, Kei; Horikawa, Tomoyasu; Kamitani, Yukiyasu

doi:10.3389/fncom.2019.00021

Cited by 96 publications

(93 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, they enhance the time resolution by adding the Markovian assumption for brain activity estimation at each time period. In Shen et al (2019) was presented a brain decoding and image reconstruction from functional magnetic resonance imaging (fMRI) activity using Deep neural networks (DNNs). The main problem with that method is the training because the size of available data is thought to be insufficient.…”

Section: Introductionmentioning

confidence: 99%

Low-Density EEG for Neural Activity Reconstruction Using Multivariate Empirical Mode Decomposition

et al. 2020

View full text Add to dashboard Cite

Several approaches can be used to estimate neural activity. The main differences between them concern the a priori information used and its sensitivity to high noise levels. Empirical mode decomposition (EMD) has been recently applied to electroencephalography EEG-based neural activity reconstruction to provide a priori time-frequency information to improve the estimation of neural activity. EMD has the specific ability to identify independent oscillatory modes in non-stationary signals with multiple oscillatory components. However, attempts to use EMD in EEG analysis have not yet provided optimal reconstructions, due to the intrinsic mode-mixing problem of EMD. Several studies have used single-channel analysis, whereas others have used multiple-channel analysis for other applications. Here, we present the results of multiple-channel analysis using multivariate empirical mode decomposition (MEMD) to reduce the mode-mixing problem and provide useful a priori time-frequency information for the reconstruction of neuronal activity using several low-density EEG electrode montages. The methods were evaluated using real and synthetic EEG data, in which the reconstructions were performed using the multiple sparse priors (MSP) algorithm with EEG electrode montages of 32, 16, and 8 electrodes. The quality of the source reconstruction was assessed using the Wasserstein metric. A comparison of the solutions without pre-processing and those after applying MEMD showed the source reconstructions to be improved using MEMD as a priori information for the low-density montages of 8 and 16 electrodes. The mean source reconstruction error on a real EEG dataset was reduced by 59.42 and 66.04% for the 8 and 16 electrode montages, respectively, and that on a simulated EEG with three active sources, by 87.31 and 31.45% for the same electrode montages.

show abstract

Section: Introductionmentioning

confidence: 99%

Low-Density EEG for Neural Activity Reconstruction Using Multivariate Empirical Mode Decomposition

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The second baseline model we trained was is generative adversarial network with a feature loss, which is based on the end-to-end reconstruction model from Shen et al [41]. We used the generator and discriminator modules present in brain2pix, however, we did not construct RFSimages for the input of the model.…”

Section: Shen Baselinementioning

confidence: 99%

Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity

Ambrogioni

Seeliger

et al. 2021

Preprint

View full text Add to dashboard Cite

Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.

show abstract

“…Reconstructing natural images from fMRI was approached by a number of methods, which can broadly be classified into three families: (i) Linear regression between fMRI data and handcrafted image-features (e.g., Gabor wavelets) [1][2][3], (ii) Linear regression between fMRI data and deep (CNN-based) image-features (e.g., using pretrained AlexNet) [4][5][6][7], or latent spaces of pretrained generative models [8][9][10][11], and (iii) End-to-end Deep Learning [12][13][14][15]. To our best knowledge, methods [6] and [13] are the current state-of-the-art in this field.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Gaziv

Beliy

Granot

et al. 2020

Preprint

View full text Add to dashboard Cite

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1

show abstract

End-to-End Deep Image Reconstruction From Human Brain Activity

Cited by 96 publications

References 22 publications

Low-Density EEG for Neural Activity Reconstruction Using Multivariate Empirical Mode Decomposition

Low-Density EEG for Neural Activity Reconstruction Using Multivariate Empirical Mode Decomposition

Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity

Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Contact Info

Product

Resources

About