Visual and Affective Grounding in Language and Mind

Deyne, Simon De; Navarro, Daniel J.; Collell, Guillem; Perfors, Amy

doi:10.31234/osf.io/q97f8

Cited by 9 publications

(19 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We expect that the multimodal model integrating linguistic, visual, and emotional information will outperform a purely linguistic model, as well as models that combine linguistic–visual and linguistic–emotional information. In addition, we expect that adding visual representations will especially be beneficial for more concrete concepts, whereas emotional information will especially be beneficial for more abstract concepts, in line with the empirical evidence reviewed above (and with initial findings from De Deyne et al, 2018). As in previous models, our work uses visual and emotional data that can only be considered as providing a static window into the embodied sensory–motor and affective states of the agent, rather than truly embodied information.…”

Section: Introductionsupporting

confidence: 73%

“…As mentioned in the introduction, a previous study (De Deyne et al, 2018) also examined the change in performance for distributional models of semantics, when adding experiential (i.e., visual and emotional) information. They found that including experiential information led to little or no improvement for internal language models, but had a moderate positive effect for external language models.…”

Section: Discussionmentioning

confidence: 99%

“…First, we avoided the potentially controversial distinction between external and internal language models. In De Deyne et al (2018), external language models derive semantic representations from corpora of language, whereas internal language models derive semantic representations from free associations. Thus, the models differ in whether they use objective or subjective data (based on a metacognitive task), but both might be argued to tap into the same construct.…”

Section: Discussionmentioning

confidence: 99%

“…The model is very different from the one by De Deyne et al (2018), which was constructed by concatenating valence, arousal, and potency ratings, for men and women separately (i.e., six dimensions), from the study by Warriner, Kuperman, and Brysbaert (2013), with valence, arousal, and dominance ratings, from the study by Mohammad (2018). DeepMoji provides better representations for our purposes than ratings because first, a model trained over a corpus of tweets, rather than subjective ratings, makes the emotion model more comparable to the linguistic and visual models, both trained over corpora.…”

Section: Methodsmentioning

confidence: 97%

“…Getting back to computational models, while many models of semantic representation have integrated linguistic and visual information, only one previous study has considered emotional information along with visual and linguistic information (De Deyne, Navarro, Collell, & Perfors, 2018). In this study, the authors compared two classes of semantic models, reflecting either external or internal language, respectively.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Constructing Semantic Models From Words, Images, and Emojis

Rotaru

Vigliocco

2020

Cognitive Science

View full text Add to dashboard Cite

A number of recent models of semantics combine linguistic information, derived from text corpora, and visual information, derived from image collections, demonstrating that the resulting multimodal models are better than either of their unimodal counterparts, in accounting for behavioral data. Empirical work on semantic processing has shown that emotion also plays an important role especially in abstract concepts; however, models integrating emotion along with linguistic and visual information are lacking. Here, we first improve on visual and affective representations, derived from state-of-the-art existing models, by choosing models that best fit available human semantic data and extending the number of concepts they cover. Crucially then, we assess whether adding affective representations (obtained from a neural network model designed to predict emojis from co-occurring text) improves the model's ability to fit semantic similarity/relatedness judgments from a purely linguistic and linguistic-visual model. We find that, given specific weights assigned to the models, adding both visual and affective representations improves performance, with visual representations providing an improvement especially for more concrete words, and affective representations improving especially the fit for more abstract words.

show abstract

Section: Introductionsupporting

confidence: 73%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 97%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Constructing Semantic Models From Words, Images, and Emojis

Rotaru

Vigliocco

2020

Cognitive Science

View full text Add to dashboard Cite

show abstract

Word meaning in minds and machines.

Lake¹,

Murphy²

2023

Psychological Review

View full text Add to dashboard Cite

Machines have achieved a broad and growing set of linguistic competencies, thanks to recent progress in Natural Language Processing (NLP). Psychologists have shown increasing interest in such models, comparing their output to psychological judgments such as similarity, association, priming, and comprehension, raising the question of whether the models could serve as psychological theories. In this article, we compare how humans and machines represent the meaning of words. We argue that contemporary NLP systems are fairly successful models of human word similarity, but they fall short in many other respects. Current models are too strongly linked to the text-based patterns in large corpora, and too weakly linked to the desires, goals, and beliefs that people express through words. Word meanings must also be grounded in perception and action and be capable of flexible combinations in ways that current systems are not. We discuss promising approaches to grounding NLP systems and argue that they will be more successful, with a more human-like, conceptual basis for word meaning.

show abstract

Similarity Judgment Within and Across Categories: A Comprehensive Model Comparison

Richie

Bhatia

2021

Cognitive Science

View full text Add to dashboard Cite

Similarity is one of the most important relations humans perceive, arguably subserving category learning and categorization, generalization and discrimination, judgment and decision making, and other cognitive functions. Researchers have proposed a wide range of representations and processes that could be at play in similarity judgment, yet have not comprehensively compared the power of these representations and processes for predicting similarity within and across different semantic categories. We performed such a comparison by pairing eight prominent vector semantic representations with seven established similarity metrics that could operate on these representations, as well as supervised methods for dimensional weighting in the similarity function. This approach yields a factorial model structure with 56 distinct representation-process pairs, which we tested on a novel dataset of similarity judgments between pairs of co-hyponymic words in eight categories. We found that cosine similarity and Pearson correlation were the overall best performing unweighted similarity functions, and that word vectors derived from free association norms often outperformed word vectors derived from text (including those specialized for similarity). Importantly, models that used human similarity judgments to learn category-specific weights on dimensions yielded substantially better predictions than all unweighted approaches across all types of similarity functions and representations, although dimension weights did not generalize well across semantic categories, suggesting strong category context effects in similarity judgment. We discuss implications of these results for cognitive modeling and natural language processing, as well as for theories of representations and processes involved in similarity.

show abstract

Visual and Affective Grounding in Language and Mind

Cited by 9 publications

References 56 publications

Constructing Semantic Models From Words, Images, and Emojis

Constructing Semantic Models From Words, Images, and Emojis

Word meaning in minds and machines.

Similarity Judgment Within and Across Categories: A Comprehensive Model Comparison

Contact Info

Product

Resources

About