Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception

Abstract

Brain-computer interfaces offer a way to communicate for people with severe motor and speech disabilities. However, successful current letter speller implementations require perception-driven paradigms (EEG) or cognitively demanding tasks (fMRI, fNIRS) which are not directly linked to letters visualized in the mind's eye. A more natural, content-based, BCI speller system immediately decoding imagined letters from associated brain activity is desirable. In the current study, we take the first steps towards such a BCI and offer new insights into the neural underpinnings of visual mental imagery, a process which is considered one of the main sources of human cognitive complexity. We demonstrate for the first time the feasibility to reconstruct visual field images which carry recognizable content of imagined letter shapes. Using submillimeter resolution fMRI

Visual mental imagery engages the left fusiform gyrus, but not the early visual cortex: a meta-analysis of neuroimaging evidence

“…The dominant model also provides a principled account of interindividual differences in VMI, because the level of activation in low-level visual areas seem to correlate with the subjective experience of "vividness" of visual mental images 22 . Recent support to this claim came from the finding that the content of VMI can be decoded from V1 activity 58 . However, in all the decoding studies visual stimuli were presented before asking subjects to imagine them.…”

Representational content in human cortical systems and visual consciousness

“…However, several considerations are also encouraging the pursuit of this goal. First of all, several successful attempts have shown that it is possible to reconstruct presented images from brain activity (Miyawaki et al, 2008;Naselaris, Prenger, Kay, Oliver, & Gallant, 2009;Nishimoto et al, 2011;Schoenmakers, Barth, Heskes, & van Gerven, 2013;Senden, Emmerling, Hoof, Frost, & Senden, 2018;Thirion et al, 2006;van Gerven, Güçlütürk, Seeliger, Ambrogioni, & Güçlü, 2018;St-Yves & Naselaris, 2019). In an important way, the task of reconstructing conscious experience from brain activity is more straightforward than that of reconstructing shown images: By most scientific and by many philosophical accounts of consciousness we should expect all information relevant for conscious experience to be available in the brain (but see O' Regan and Noë, 2001).…”

“…By contrast, information about image details are incomplete given that not all aspects of an image are necessarily encoded by our senses (St-Yves & Naselaris, 2019); these aspects of an image can never be recovered from brain activity. Furthermore, a few studies have shown that it is possible to reconstruct imagined images (Naselaris, Olman, Stansbury, Ugurbil, & Gallant, 2015) and letters (Senden et al, 2018) from brain activity. This is particularly encouraging for reconstruction of conscious experience since during imagery no sensory information from the stimulus was available (thus bypassing the demarcation problem).…”

“…All of the above studies collected independent data that could be used to fit the parameters of the encoding model to every individual participant. For reconstruction, the mapping then needs to be inverted such that it points from brain response patterns to image space, for which different solutions exist (Naselaris et al, 2009;Rezende, Mohamed, & Wierstra, 2014;Senden et al, 2018;St-Yves & Naselaris, 2019).…”

“…An interesting question is whether these brain responses should be used for fitting parameters of the encoding model (step A) or during reconstruction (step B) (or both). For letter and image imagery, parameters were fitted using data from conventional presentation of letters or natural images Senden et al, 2018) and only reconstruction was based on imagery data. This approach relies on the assumption that neural processes are shared between perception and imagery, which is supported by evidence (Pearson, Naselaris, Holmes, & Kosslyn, 2015).…”