Images of the unseen: extrapolating visual representations for abstract and concrete words in a data-driven computational model

Abstract

Theories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants’ judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don’t have.

ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation

“…used vision-based representations to demonstrate perception-based conceptual combination effects during the processing of compound words such as swordfish. In another study, Günther, Petilli, Vergallito, and Marelli (2020) found that vision-based representations can, with some success, be predicted from text-based distributional vectors, offering a mechanism via which non-experienced objects can be grounded in visual experience. Similarly, Lazaridou et al (2017) demonstrated that visual intuitions for novel words learned from text alone can be predicted from their model combining textual with visual representations.…”

“…In the present article, we present the ViSpa (Vision Spaces) system, an adaptation of the VGG-F model that, in addition to representations for individual images, includes prototypical vision-based representations for concepts. Precursors of ViSpa were already employed in some of the studies outlined in the previous section Günther, Petilli, Vergallito, & Marelli, 2020;Petilli et al, 2021).…”

“…As described in the previous section, all individual images in the IMG space are associated to category labels. This allows us to calculate prototypical representations for categories as the centroid of the 100 to 200 individual vision-based vectors extracted for each category Günther, Petilli, Vergallito, & Marelli, 2020;Petilli et al, 2021; see also Anderson, Bruni, Lopopolo, Poesio, & Baroni, 2015). This was done for all layers of the network, thus resulting in nine different prototype (PRO) spaces.…”

“…Even though the pioneering results reviewed earlier speak for DCNNs as promising candidates for modeling many aspects of human cognition (Jozwik et al, 2017), a thorough large-scale evaluation in this respect is still pending. Such an evaluation is however urgently needed if one aims to employ these models to investigate theoretical questions (see Günther, Petilli, Vergallito, & Marelli, 2020;Lazaridou et al, 2015Lazaridou et al, , 2017. Rating studies have focused on pairs of individual images (Jozwik et al, 2017;Zhang et al, 2018) and were based on relatively limited sets of images (around 100).…”

Emotional Valence Precedes Semantic Maturation of Words: A Longitudinal Computational Study of Early Verbal Emotional Anchoring

“…Theoretical proposals like the language and situated simulation theory (Barsalou, Santos, Simmons, & Wilson, 2008) or the symbol interdependency hypothesis (Louwerse, 2011(Louwerse, , 2018 have been developed to explain how these amodal representations can generate emotional and other sensorimotor word representations. Additionally, some mapping mechanisms have also been proposed in vector space models to act as a link between amodal and modal representations Günther, Petilli, Vergallito, & Marelli, 2020;Hollis, Westbury, & Lefsrud, 2017;Martínez-Huertas, Jorge-Botana, Luzón, & Olmos, 2021). In some contexts, this perspective has been termed the specific dimensionality hypothesis, as only some parts of the amodal representation seem to be mapping both formats of representation.…”