A Distributed Network for Multimodal Experiential Representation of Concepts

Abstract

The architecture of the cortical system underlying concept representation is a topic of intense debate. Much evidence supports the claim that concept retrieval selectively engages sensory, motor, and other neural systems involved in the acquisition of the retrieved concept, yet there is also strong evidence for involvement of high-level, supramodal cortical regions. A fundamental question about the organization of this system is whether modality-specific information originating from sensory and motor areas is integrated across multiple "convergence zones" or in a single centralized "hub". We used representational similarity analysis (RSA) of fMRI data to map brain regions where the similarity structure of neural patterns elicited by large sets of concepts matched the similarity structure predicted by a high-dimensional model of concept representation based on sensory, motor, affective, and other modal aspects of experience.Across two studies involving different sets of concepts, different participants, and different tasks, searchlight RSA revealed a distributed, bihemispheric network engaged in multimodal experiential representation, composed of high-level association cortex in anterior, lateral, and ventral temporal lobe; inferior parietal lobule; posterior cingulate gyrus and precuneus; and medial, dorsal, ventrolateral, and orbital prefrontal cortex. These regions closely resemble networks previously implicated in general semantic and "default mode" processing and are known to be high-level hubs for convergence of multimodal processing streams. Supplemented by an exploratory cluster analysis, these results indicate that the concept representation system consists of multiple, hierarchically organized convergence zones supporting multimodal integration of experiential information. Significance StatementIt has long been known that information about visual, auditory, motor, affective, and other features of our phenomenal experience originate in distinct brain regions. However, it is still unclear how these processing streams converge to form multimodal concept representations..

Meta-analytic evidence for a novel hierarchical model of conceptual processing

“…MVPA-including decoding and representational similarity analysis (RSA)-tests for information represented in fine-grained, multi-voxel activity patterns (Haxby et al, 2014;Norman et al, 2006). RSA has recently been used to relate computational models of semantics to the brain, which revealed that a grounded perceptual-motor model better explains brain representations (including in multimodal regions) than taxonomic categories or distributional information (Fernandino et al, 2022;Tong et al, 2022). These findings clearly corroborate our results and our model.…”

“…While such "cross-modality spreading" cannot be completely excluded, it is unlikely to explain all multimodal activations, especially in the trimodal IPL and pMTG. Individual studies found multimodal effects in left IPL and pMTG, even when the individual modalities were controlled for (Kuhnke et al, 2020b;Tong et al, 2022). Many experiments included in this meta-analysis similarly isolated modality-specific activity, while controlling for other modalities (e.g.…”

The Primacy of Experience in Language Processing: Semantic Priming Is Driven Primarily by Experiential Similarity

“…We had previously shown that individual words could be decoded from fMRI activation patterns in these areas using a multimodal sensory-motor model, but not with a model based on ortho-phonological features of the corresponding word forms (Fernandino, 2016b). Furthermore, the similarity structure of fMRI activation patterns in these regions predicts the semantic similarity structure of both object and event nouns, and it does so significantly more accurately when semantic similarity is estimated based on experiential features than when it is based on taxonomic or distributional information (Fernandino et al, 2022;Tong et al, 2022).…”

Conceptual representations in the default, control and attention networks are task-dependent and cross-modal

“…For instance, action features are represented in somatomotor regions (Hauk et al, 2004;Tettamanti et al, 2005;Vukovic et al, 2017), while sound features are represented in auditory areas (Bonner and Grossman, 2012;Kiefer et al, , 2008Trumpp et al, 2013). Cross-modal convergence zones integrate modality-specific features into more abstract, cross-modal representations (Binder, 2016;Fernandino et al, 2016a;Kuhnke et al, 2023Kuhnke et al, , 2020bTong et al, 2022). We previously proposed a distinction among cross-modal convergence zones between "multimodal" regions which retain modality-specific information, and "amodal" regions which completely abstract away from modality-specific input (Kuhnke et al, 2023(Kuhnke et al, , 2022(Kuhnke et al, , 2020b.…”