Modality-independent coding of scene categories in prefrontal cortex

Abstract

Natural environments convey information through multiple sensory modalities, all of which contribute to people's percepts. Although it has been shown that neural representations of visual content can be decoded from the visual cortex, it remains unclear where and how humans represent perceptual information at a conceptual level, not limited to a specific sensory modality. To address this question, we investigated how categories of scene images and sounds are represented in several brain regions. We found that both visual and auditory scene categories can be decoded not only from modality-specific areas, but also from several brain regions in the temporal, parietal, and prefrontal cortex. Intriguingly, only in the prefrontal cortex, but not in any other regions, categories of scene images and sounds appear to be represented in similar activation patterns, suggesting that scene representations in prefrontal cortex are modality-independent. Furthermore, the error patterns of neural decoders indicate that the category-specific neural activity patterns in the middle and superior frontal gyri are tightly linked to categorization behavior. Our findings suggest that complex visual information is represented at a conceptual level in prefrontal cortex, regardless of the sensory modality of the stimulus.

Spatiotemporal Dynamics of Sound Representations reveal a Hierarchical Progression of Category Selectivity

“…3B,D). Traditionally sensory-specific areas have long been implicated in cross-sensory processing (Jung et al, 2018;Kim and Zatorre, 2011;Kriegstein et al, 2005;Smith and Goodale, 2015;Vetter et al, 2014); however, in this case, the regions were activated in the absence of any visual stimulation or association with a visual cue, ruling out a multisensory integrative or paired associative process. Auditory objects and scenes have resisted easy analogical transfer from their visual counterparts, both as fundamental concepts (Griffiths and Warren, 2004) and as variables in experiments, due to differences in spatiotemporal structure of an image vs. a sound .…”

Task dependence of neural representations of scene attributes but not scene categories in the prefrontal cortex

“…As we identified the neural representations of temperature and sound level in scene images, we here further explore these representations to better understand how scene content is represented in PFC. Previous research has demonstrated that the prefrontal cortex represents scene categories at an abstract-level, which generalizes between visual and auditory input (Jung et al, 2018). Thus, it is possible that the prefrontal cortex represents global scene properties, temperature and sound-level information, at such an abstract level as well.…”

“…Research has demonstrated that various components of scenes are processed across several brain regions of the scene-selective network in the visual cortex, such as the parahippocampal place area (PPA, Epstein & Kanwisher, 1998), the retrosplenial cortex (RSC, Maguire, 2001), and the occipital place area (OPA, Dilks et al, 2013). However, recent studies have shown that visual scene processing takes place beyond the visual cortex and involves the associative cortex such as the parietal (Silson et al, 2016; Silson et al, 2019) and the prefrontal cortex (Jung et al, 2018). Does this indicate that the scene processing network can be extended to the prefrontal cortex (PFC)?…”

“…We have previously demonstrated that scene images and sounds elicit neural representations of scene categories in PFC. Intriguingly, these representations are modality-independent, suggesting that PFC represents scene content in an abstract way (Jung et al, 2018). Natural scenes also comprise other attributes, often referred to as global scene properties, such as openness, temperature, or transience (Greene & Oliva, 2009), which are also processed in scene-selective visual brain regions (Kravitz et al, 2011; Park et al, 2011; Persichetti and Dilks, 2019).…”

“…First, behavioral studies have shown that scene category information is accessible with limited attention (Li et al, 2002; Mack & Rock, 1998), and even when it is not relevant to the task context (Greene & Fei-Fei, 2014). Second, it has been shown that scene categories can be represented in PFC in a passive-viewing paradigm (Jung et al, 2018; Walther et al, 2009), which does not explicitly require attention to scene categories. Thus, we predict that scene category information may be represented in PFC regardless of task context.…”