The grounding of abstract concepts in the motor and visual system: An fMRI study

Harpaintner, Marcel; Sim, Eun-Jin; Trumpp, Natalie M.; Martin, Ulrich; Kiefer, Markus

doi:10.1016/j.cortex.2019.10.014

Cited by 81 publications

(87 citation statements)

References 153 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The acquired knowledge is stored in cerebral cortex as semantic concept representations, which support a range of cognitive processes including language understanding. Many previous fMRI studies have found that concepts are represented near the perceptual systems through which they are commonly experienced (Binder and Desai, 2011;Harpaintner et al, 2020;Martin, 2016). These studies support grounded cognition theories, which hold that a concept's semantic representation is formed through generalization or re-enactment of perceptual representations involved in learning the concept (Barsalou, 2008;Binder and Desai, 2011).…”

Section: Introductionsupporting

confidence: 70%

Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

Tang

LeBel

Huth

2021

Preprint

View full text Add to dashboard Cite

The human semantic system stores knowledge acquired through both perception and language. To study how semantic representations in cortex integrate perceptual and linguistic information, we created semantic word embedding spaces that combine models of visual and linguistic processing. We then used these visually-grounded semantic spaces to fit voxelwise encoding models to fMRI data collected while subjects listened to hours of narrative stories. We found that cortical regions near the visual system represent concepts by combining visual and linguistic information, while regions near the language system represent concepts using mostly linguistic information. Assessing individual representations near visual cortex, we found that more concrete concepts contain more visual information, while even abstract concepts contain some amount of visual information from associated concrete concepts. Finally we found that these visual grounding effects are localized near visual cortex, suggesting that semantic representations specifically reflect the modality of adjacent perceptual systems. Our results provide a computational account of how visual and linguistic information are combined to represent concrete and abstract concepts across cortex.

show abstract

Section: Introductionsupporting

confidence: 70%

Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

Tang

LeBel

Huth

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Mean correct latencies were analysed by means of a one-sided paired-samples t-test testing the same directional hypothesis as Kuipers et al, (2018). We conducted analyses within a Bayesian framework relying on prior specification of theoretical knowledge which includes effect sizes and the direction of hypotheses [33], see also [41]. The analysis revealed an effect of Target Word Type with shorter latencies for the related pairs than the unrelated ones (M Related = 1411 ms; SD = 429; M Non_Related = 1529 ms; SD = 464; Mean Difference = 118 ms; 95CI [21,214]; t(19) = 2.55, p = 0.01, Cohen's d = 0.57, see Fig 2).…”

Section: Resultsmentioning

confidence: 99%

Intangible features extraction in the processing of abstract concepts: Evidence from picture-word priming

2021

View full text Add to dashboard Cite

Over the last decade, hypotheses ranging from linguistic symbol processing to embodiment have been formulated to account for the content and mechanisms responsible for the representation of abstract concepts. Results of recent studies have suggested that abstract concepts, just like concrete ones, can benefit from knowledge of real-world situational context, but that they can also be processed based on abstract pictures devoid of such situational features. This paper presents two semantic priming experiments to explore such mechanisms further. The first experiment replicates Kuipers, Jones, and Thierry (2018) in a cross-linguistic setting which shows that abstract concepts can be processed from abstract pictures devoid of tangible features. In the second experiment, we studied extraction mechanisms that come into play when participants are presented with abstract and concrete pictures that provide situational information to illustrate target abstract concepts. We expected this facilitatory effect to be limited to concrete picture primes. Our data analysed with both Bayesian and Frequentist tests showed however that even when presented with tangible situational information, the extraction of features still occurred for abstract pictures. We discuss the implications of this with respect to future avenues for studying the processing of abstract concepts.

show abstract

“…For example, Harpaintner et al (2018) identify different subgroups of abstract concepts, which are related to sensorimotor information (visual: observation, gustatory: appetite, interoceptive: pain, etc. ), social constellations (advice), emotions/internal states (emotion), or verbal associations (revolution), with processing differences between these groups (Harpaintner et al, 2020). Since our model operates on visual information, it stands to reason that model performance might be higher for words related to sensorimotor information and especially visual information as compared to the other subgroups.…”

Section: Discussionmentioning

confidence: 99%

“…Over the recent years many different proposals have been made addressing this issue (see Barsalou, Santos, Simmons, & Wilson, 2008;Borghi & Binkofski, 2014;Harpaintner, Trumpp, & Kiefer, 2018;Harpaintner, Sim, Trumpp, Ulrich, & Kiefer, 2020;Hoffman, McClelland, & Lambon Ralph, 2018;Kousta, Vigliocco, Vinson, Andrews, & Del Campo, 2011;Lakoff & Johnson, 2008;Wilson-Mendenhall, Simmons, Martin, & Barsalou, 2013, for different theoretical approaches). One possible mechanism of how grounding can be established in the absence of experience (referred to as acquired embodiment by Hoffman et al, 2018 and indirect grounding by Günther, Nguyen, et al, 2020) is best illustrated by an example: Assume a friend tells you, "On my way here, I saw a little wibby chirping in a tree!".…”

mentioning

confidence: 99%

Images of the unseen: Extrapolating visual representations for abstract and concrete words in a data-driven computational model

Guenther¹,

Petilli²,

Vergallito³

et al. 2020

Preprint

View full text Add to dashboard Cite

Theories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants' judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don't have.

show abstract

The grounding of abstract concepts in the motor and visual system: An fMRI study

Cited by 81 publications

References 153 publications

Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

Intangible features extraction in the processing of abstract concepts: Evidence from picture-word priming

Images of the unseen: Extrapolating visual representations for abstract and concrete words in a data-driven computational model

Contact Info

Product

Resources

About