Real-World Scene Representations in High-Level Visual Cortex: It's the Spaces More Than the Places

Abstract

Real-world scenes are incredibly complex and heterogeneous, yet we are able to identify and categorize them effortlessly. In humans, the ventral temporal Parahippocampal Place Area (PPA) has been implicated in scene processing, but scene information is contained in many visual areas, leaving their specific contributions unclear. While early theories of PPA emphasized its role in spatial processing, more recent reports of its function have emphasized semantic or contextual processing. Here, using functional imaging, we reconstructed the organization of scene representations across human ventral visual cortex by analyzing the distributed response to 96 diverse real-world scenes. We found that while individual scenes could be decoded in both PPA and early visual cortex (EVC), the structure of representations in these regions was vastly different. In both regions spatial rather than semantic factors defined the structure of representations. However, in PPA, representations were defined primarily by the spatial factor of expanse (open, closed) and in EVC primarily by distance (near, far). Further, independent behavioral ratings of expanse and distance correlated strongly with representations in PPA and pEVC, respectively. In neither region was content (manmade, natural) a major contributor to the overall organization. Further, the response of PPA could not be used to decode the high-level semantic category of scenes even when spatial factors were held constant, nor could category be decoded across different distances. These findings demonstrate, contrary to recent reports, that the response PPA primarily reflects spatial, not categorical or contextual aspects of real-world scenes.

Evidencing a place for the hippocampus within the core scene processing network

“…This core network is thought to include posterior parahippocampal gyrus [PHG; Aguirre et al, 1998a; Epstein et al, 2003; Epstein and Kanwisher, 1998], retrosplenial cortex [RSC; Auger et al, 2012; Epstein et al, 2007; Vann et al, 2009] and the transverse occipital sulcus [TOS; Dilks et al, 2013; Ganaden et al, 2013; He et al, 2013; Mullin and Steeves, 2011; Nasr et al, 2011]. As seen in the neural processing of other visual categories [Taylor and Downing, 2011], these regions appear to support distinct but complementary aspects of scene processing, and are differentially modulated by changes in viewpoint [Epstein et al, 2003, 2007; Park and Chun, 2009], spatial layout [Harel et al, 2013; Park et al, 2015], and lower‐level spatial features [Kravitz et al, 2011a; Nasr et al, 2014]. Importantly, while functional differences exist between these different regions, they all share the critical property of showing a preferential response to scenes/places.…”

Similarity representation of pattern-information fMRI

“…Direct visual inspection of RDMs may yield clustered structures in neural response patterns [22,23]. Nevertheless, the dissimilarity structures can be measured and analyzed by automated algorithms of multi-dimensional scaling (MDS) [24] and hierarchical clustering analysis [25].…”

“…It is not until recently that applications of this method have gained momentum and developed rapidly. In 2011, Kravitz et al [23] applied MDS to reconstruct and visualize representational dissimilarity structure derived from the distributed neural responses to 96 diverse real-world scenes. The authors found, contrary to previous studies [29,30], that representations in the ventral temporal parahippocampal place area (PPA) were characterized primarily by the spatial factor of expanse (open, closed) and in early visual cortex (EVC) primarily by distance (near, far), not by category or context.…”

“…To investigate this question demands across-subject decoding, where visualizing and comparing neural similarity space by MDS and hierarchical clustering analysis are commonly used [23,35]. However, classification by intuitive visualization or averaging alone might not be self-evident at a finer grained scale and not sufficient to judge whether an attempt to decode the stimuli across subjects succeeds or fails.…”

Auditory and visual scene analysis: an overview

“…While early work on auditory and visual scene analysis focused on relatively simple artificial scenes, recently research has been extended to more realistic situations, such as simulated cocktail parties [10,11] and natural visual scenes [12][13][14]. Furthermore, scene analysis is facilitated by the use of statistical regularities.…”