Modality-Independent Coding of Scene Categories in Prefrontal Cortex

Abstract

Natural environments convey information through multiple sensory modalities, all of which contribute to people's percepts. Although it has been shown that visual or auditory content of scene categories can be decoded from brain activity, it remains unclear how humans represent scene information beyond a specific sensory modality domain. To address this question, we investigated how categories of scene images and sounds are represented in several brain regions. A group of healthy human subjects (both sexes) participated in the present study, where their brain activity was measured with fMRI while viewing images or listening to sounds of different real-world environments. We found that both visual and auditory scene categories can be decoded not only from modality-specific areas, but also from several brain regions in the temporal, parietal, and prefrontal cortex (PFC). Intriguingly, only in the PFC, but not in any other regions, categories of scene images and sounds appear to be represented in similar activation patterns, suggesting that scene representations in PFC are modality-independent. Furthermore, the error patterns of neural decoders indicate that category-specific neural activity patterns in the middle and superior frontal gyri are tightly linked to categorization behavior. Our findings demonstrate that complex scene information is represented at an abstract level in the PFC, regardless of the sensory modality of the stimulus. Our experience in daily life includes multiple sensory inputs, such as images, sounds, or scents from the surroundings, which all contribute to our understanding of the environment. Here, for the first time, we investigated where and how in the brain information about the natural environment from multiple senses is merged to form modality-independent representations of scene categories. We show direct decoding of scene categories across sensory modalities from patterns of neural activity in the prefrontal cortex (PFC). We also conclusively tie these neural representations to human categorization behavior by comparing patterns of errors between a neural decoder and behavior. Our findings suggest that PFC is a central hub for integrating sensory information and computing modality-independent representations of scene categories.

Tasting Pictures: Viewing Images of Foods Evokes Taste-Quality-Specific Activity in Gustatory Insular Cortex

“…Further analysis of these results indicated that not only were the multivariate patterns produced by tastes and pictures completely unrelated, but the decoding models relied most heavily on independent sets of voxels when classifying food pictures or tastes. These results are partly in keeping with previous studies which have generated similar null results when attempting cross-modal classification within primary sensory cortices (24,25).…”

“…These results demonstrate how higher-order inferences derived from stimuli in one modality (in this case vision) can be represented in brain regions typically thought to represent only low-level information about a different modality (in this case taste). Broadly, these results echo previous neuroimaging findings on multisensory processing within vision and audition, which demonstrate that early sensory cortical areas are able to represent the inferred sensory properties of stimuli presented via another sensory channel (24,25). Taken together, these findings are consistent with claims that both higher-order and presumptively unimodal areas of neocortex are fundamentally multisensory in nature (26).…”

Where Sounds Occur Matters: Context Effects Influence Processing of Salient Vocalisations

“…Evaluation of sounds together with contextual information also falls within the purview of the PFC. In an fMRI study which showed the context-dependent nature of sound processing, [26] used 640 images from four scene categories (beach, office, city, forest) and 64 sound clips that corresponded to the same scenes (e.g., sound of waves on beach, people talking in the office) to discover that, when the two modalities are congruent to the same scene category, a similar neural activation pattern in response to visual and auditory stimuli occurs only in the PFC, suggesting its critical role in coordinating abstract visual and auditory scenes relative to context. Neural activation patterns were found to be distinct in different categories of scenes, indicating that contextual interpretations of scenes may fall within the purview of the PFC.…”

Task dependence of neural representations of scene attributes but not scene categories in the prefrontal cortex

“…Research has demonstrated that various components of scenes are processed across several brain regions of the sceneselective network in the visual cortex, such as the parahippocampal place area (PPA, Epstein & Kanwisher, 1998), the retrosplenial cortex (RSC, Maguire, 2001), and the occipital place area (OPA, Dilks et al, 2013). However, recent studies have shown that visual scene processing takes place beyond the visual cortex and involves the associative cortex such as the parietal (Silson et al, 2016;Silson et al, 2019) and the prefrontal cortex (Jung et al, 2018). Does this indicate that the scene processing network can be extended to the prefrontal cortex (PFC)?…”

“…We have previously demonstrated that scene images and sounds elicit neural representations of scene categories in PFC. Intriguingly, these representations are modality-independent, suggesting that PFC represents scene content in an abstract way (Jung et al, 2018). Natural scenes also comprise other attributes, often referred to as global scene properties, such as openness, temperature, or transience (Greene & Oliva, 2009), which are also processed in scene-selective visual brain regions (Kravitz et al, 2011;Park et al, 2011;Persichetti and Dilks, 2019).…”