A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes

Abstract

Recent multi-voxel pattern classification (MVPC) studies have shown that in early visual cortex patterns of brain activity generated during mental imagery are similar to patterns of activity generated during perception. This finding implies that low-level visual features (e.g., space, spatial frequency, and orientation) are encoded during mental imagery. However, the specific hypothesis that low-level visual features are encoded during mental imagery is difficult to directly test using MVPC. The difficulty is especially acute when considering the representation of complex, multi-object scenes that can evoke multiple sources of variation that are distinct from low-level visual features. Therefore, we used a voxel-wise modeling and decoding approach to directly test the hypothesis that low-level visual features are encoded in activity generated during mental imagery of complex scenes. Using fMRI measurements of cortical activity evoked by viewing photographs, we constructed voxel-wise encoding models of tuning to low-level visual features. We also measured activity as subjects imagined previously memorized works of art. We then used the encoding models to determine if putative low-level visual features encoded in this activity could pick out the imagined artwork from among thousands of other randomly selected images. We show that mental images can be accurately identified in this way; moreover, mental image identification accuracy depends upon the degree of tuning to low-level visual features in the voxels selected for decoding. These results directly confirm the hypothesis that low-level visual features are encoded during mental imagery of complex scenes. Our work also points to novel forms of brain-machine interaction: we provide a proof-of-concept demonstration of an internet image search guided by mental imagery.

Domain-specific reports of visual imagery vividness are not related to perceptual expertise

“…Most evidence of visual imagery has come from imaging techniques, demonstrating that brain regions active during the perception of visual stimuli also show increased activation during visual imagery (Goebel, Khorram-Sefat, Muckli, Hacker, & Singer, 1998;Johnson & Johnson, 2014;Kosslyn, Thompson, & Alpert, 1997;Lee, Kravitz, & Baker, 2012;O'Craven & Kanwisher, 2000;Reddy, Tsuchiya, & Serre, 2010;Stokes, Thompson, Cusack, & Duncan, 2009). Moreover, visual stimuli can be reliably decoded from the activity patterns in early visual regions (Albers, Kok, Toni, Dijkerman, & de Lange, 2013;Naselaris, Olman, Stansbury, Ugurbil, & Gallant, 2015). Neuronal recordings also support the idea that perception and imagery overlap (Kreiman, Koch, & Fried, 2000).…”

Distributed Patterns of Reactivation Predict Vividness of Recollection

“…These visualizations may or may not have been modeled on perception. For example, it may be that mental images were not projected reliably into the same retinotopic frame in which they were processed during direct perception (but see Naselaris, Olman, Stansbury, Ugurbil, & Gallant, 2015). Our results provide some evidence of the visual cortex's participation in the reactivation of vivid memories, and future work should attempt to elucidate the nature of this contribution.…”

The heterogeneity of mental representation: Ending the imagery debate

“…For example, it is possible that a nondepictive commonality (e.g., focus of attention, expectation of reward) between imagery and perception is driving the algorithm's decoding success. However, for the first time to our knowledge, a recent study was able to overcome this ambiguity by explicitly using a sensory multifeature-based encoding model (13). The authors fit a voxel-wise Gabor wavelet model to each voxel based on the blood oxygen level-dependent activity triggered by complex perceptual images.…”