Connecting concepts in the brain by mapping cortical representations of semantic relations

Abstract

In the brain, the semantic system is thought to store concepts. However, little is known about how it connects different concepts and infers semantic relations. To address this question, we collected hours of functional magnetic resonance imaging data from human subjects listening to natural stories. We developed a predictive model of the voxel-wise response and further applied it to thousands of new words. Our results suggest that both semantic categories and relations are represented by spatially overlapping cortical patterns, instead of anatomically segregated regions. Semantic relations that reflect conceptual progression from concreteness to abstractness are represented by cortical patterns of activation in the default mode network and deactivation in the frontoparietal attention network. We conclude that the human brain uses distributed networks to encode not only concepts but also relationships between concepts. In particular, the default mode network plays a central role in semantic processing for abstraction of concepts.

Cortical network responses map onto data-driven features that capture visual semantics of movie fragments

“…But, rather than addressing hypotheses by formulating specific questions and constraining the experimental paradigm, the current bottom-up analysis of data obtained in a natural context maps deep structures of visual input onto neural activity. Instead of looking to decode pre-selected discrete semantic categories we focused on the variance along individual orthogonal semantic axes 37,39,83 extracted from the raw input itself. The fact that the categories that have traditionally been investigated, such as faces, places, movement and body parts turned out to be the principal components in the visual domain of the feature film, underscores the usefulness of a more holistic approach for investigating neural substrates of attribution of semantic meaning to visual input.…”

“…Language models that extract semantic properties of words by learning their co-occurrence patterns have also been successfully related to neural data through either encoding 36 or decoding 54 , 55 approaches. Exploiting the co-occurrence patterns to study encoding of meaning in the brain has proven fruitful regardless of whether the co-occurrence patterns were extracted through an artificial neural network 36 , 37 , 55 or simpler corpus-based approaches 35 , 56 , underlining the importance of contextual information in semantic representation.…”

“…Multiple studies showed that these models can fit evoked cortical responses better than alternative representations of stimuli typically based on hand-engineered filters for low-level perceptual input, for example, Gabor filters for pixel or sound spectrogram input [32][33][34] . Artificial neural network approaches to modeling language have also shown to correlate well with the cortical responses to language related stimuli [35][36][37] .…”

l-Lactate: Food for Thoughts, Memory and Behavior

“…According to classical literature, for semantic memory, the main brain area involved is the perirhinal cortex [34,35,37,41]. More recent studies have shown, particularly analyzing PET scans and fMRI images, reveal that semantic memory is represented by spatially overlapping cortical patterns rather than anatomically segregated regions [42]. Binder and co-workers performed a meta-analysis of 120 studies and concluded that semantic processing occupies a large portion of the cortex and that it could be divided in three broad categories: Posterior heteromodal association cortex (posterior inferior parietal lobe, middle temporal gyrus and fusiform gyrus), subregions of the heteromodal prefrontal cortex (dorsal, inferior, ventromedial prefrontal cortex) and medial paralimbic regions (parahippocampus and posterior cingulate gyrus) [43].…”

Relation Classification with Cognitive Attention Supervision

“…For humans, the task of determining semantic relationships may entail complicated inference based on concepts' contexts (Yee and Thompson-Schill, 2016;Zhang et al, 2020) and commonsense knowledge (e.g., causal relations; Chiang et al, 2021), and for labeling relations between entities in texts the task may depend on the genre of the text (e.g. biomedical, biographical) and constraints indicated by annotator instructions (Mohammad, 2016).…”