Bayesian Reconstruction of Natural Images from Human Brain Activity

Abstract

Summary Recent studies have used fMRI signals from early visual areas to reconstruct simple geometric patterns. Here, we demonstrate a new Bayesian decoder that uses fMRI signals from early and anterior visual areas to reconstruct complex natural images. Our decoder combines three elements: a structural encoding model that characterizes responses in early visual areas; a semantic encoding model that characterizes responses in anterior visual areas; and prior information about the structure and semantic content of natural images. By combining all these elements, the decoder produces reconstructions that accurately reflect both the spatial structure and semantic category of the objects contained in the observed natural image. Our results show that prior information has a substantial effect on the quality of natural image reconstructions. We also demonstrate that much of the variance in the responses of anterior visual areas to complex natural images is explained by the semantic category of the image alone.

Progress and challenges in probing the human brain

“…Since its introduction more than a decade ago, MVPA has been used in a number of domains to demonstrate the predictive ability of fMRI activation patterns. Perhaps the most impressive are demonstrations of the ability to successful reconstruct visual scenes 19 and faces 20 from BOLD activity patterns; similar advances have been made for higher cognitive functions such as word meaning 21 . These studies go beyond simply differentiating between experimental conditions, as they show how the underlying representational spaces relate to brain activity; for example, Huth and colleagues 22 developed a model that estimated the response at each location on the cortical surface to a large number of visual and semantic features present in natural movies (Fig.…”

Word meaning in the ventral visual path: a perceptual to conceptual gradient of semantic coding

“…Researchers capitalizing from both machine learning techniques and Representational Similarity Analysis (RSA) frameworks have shown that it is possible to discriminate between words belonging to different semantic categories (e.g., animals vs tools) as well as sub-categorical clusters (e.g., mammals vs insects) using distributed patterns of brain activation (Shinkareva et al, 2011;Bruffaerts et al, 2013;Devereux et al, 2013;Fairhall and Caramazza, 2013;Simanova et al, 2014), but they did not determine if such discriminations were driven by conceptual or/and by correlated perceptual information (Naselaris and Kay, 2015). Finally, the so called "encoding" approach (modelling and predicting voxel-wise activation for different stimuli according to their defining set of features) has been successfully applied to predict brain activation during the elaboration of images and movies (Naselaris et al, 2009;Nishimoto et al, 2011), and only very recently to words (Fernandino et al, 2015a). This last study, despite being similar to the present research in that it investigate the semantic coding of symbols (words), only investigated the impact of what we call here "perceptual features"…”

Tempo de cérebro

“…Hoje é possível classificar estados psíquicos utilizando dados de imageamento cerebral, isto é, tornou-se possível ler mentes. Isso foi demonstrado para imagens detectadas no córtex visual (Naselaris et al, 2009;Kay et al, 2008). Resultados semelhantes e ainda mais impressionantes foram obtidos em todo o córtex cerebral em indivíduos engajados apenas na mentalização de palavras (Mitchell et al, 2008).…”