Semantic projection recovers rich human knowledge of multiple object features from word embeddings

Grand, Gabriel; Blank, Idan; Pereira, Francisco; Fedorenko, Evelina

doi:10.1038/s41562-022-01316-8

Cited by 65 publications

(70 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To generate predictions for object feature ratings using embedding spaces, we adapted and extended a previously used vector projection method first employed by Grand et al (2018) and Richie et al (2019). These prior approaches manually defined three separate adjectives for each extreme end of a particular feature (e.g., for the "size" feature, adjectives representing the low end are "small," "tiny," and "minuscule," and adjectives representing the high end are "large," "huge," and "giant").…”

Section: Contextual Projection: Defining Feature Vectors In Embedding...mentioning

confidence: 99%

“…(vehicles not in the testing set). By contrast, prior work using projection techniques to predict feature ratings from embedding spaces (Grand et al, 2018;Richie et al, 2019) has used adjectives as endpoints, ignoring the potential influence of domain-level semantic context on similarity judgments (e.g., "size" was defined as a vector from "small," "tiny," "minuscule" to "large," "huge," "giant," regardless of semantic context). However, as we argued above, feature ratings may be impacted by semantic context much as-and perhaps for the same reasons as-similarity judgments.…”

Section: Experiments 2: Contextual Projection Captures Reliable Infor...mentioning

confidence: 99%

“…This approach has met with some success in predicting categorical distinctions (Baroni, Dinu, & Kruszewski, 2014), predicting properties of objects (Grand, Blank, Pereira, & Fedorenko, 2018; Pereira, Gershman, Ritter, & Botvinick, 2016; Richie et al., 2019), and even revealing cultural stereotypes and implicit associations hidden within the documents (Caliskan et al., 2017). However, the spaces generated by such machine learning methods have remained limited in their ability to predict direct empirical measurements of human similarity judgments (Mikolov, Yih, et al., 2013; Pereira et al., 2016) and feature ratings (Grand et al., 2018). Nevertheless, this work suggests that the multidimensional representations of relationships between words (i.e., word vectors) can be used as a methodological scaffold to describe and quantify the structure of semantic knowledge and, as such, can be used to predict empirical human judgments.…”

Section: Introductionmentioning

confidence: 99%

“…A distance metric between a given pair of word vectors can then be used as a measure of their similarity. This approach has met with some success in predicting categorical distinctions (Baroni, Dinu, & Kruszewski, 2014), predicting properties of objects (Grand, Blank, Pereira, & Fedorenko, 2018; Pereira, Gershman, Ritter, & Botvinick, 2016; Richie et al., 2019), and even revealing cultural stereotypes and implicit associations hidden within the documents (Caliskan et al., 2017). However, the spaces generated by such machine learning methods have remained limited in their ability to predict direct empirical measurements of human similarity judgments (Mikolov, Yih, et al., 2013; Pereira et al., 2016) and feature ratings (Grand et al., 2018).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large‐Scale Text Corpora

et al. 2022

View full text Add to dashboard Cite

Applying machine learning algorithms to automatically infer relationships between concepts from large-scale collections of documents presents a unique opportunity to investigate at scale how human semantic knowledge is organized, how people use it to make fundamental judgments ("How similar are cats and bears?"), and how these judgments depend on the features that describe concepts (e.g., size, furriness). However, efforts to date have exhibited a substantial discrepancy between algorithm predictions and human empirical judgments. Here, we introduce a novel approach to generating embeddings for this purpose motivated by the idea that semantic context plays a critical role in human judgment. We leverage this idea by constraining the topic or domain from which documents used for generating embeddings are drawn (e.g., referring to the natural world vs. transportation apparatus). Specifically, we trained state-of-the-art machine learning algorithms using contextually-constrained text corpora (domain-specific subsets of Wikipedia articles, 50+ million words each) and showed that this procedure greatly improved predictions of empirical similarity judgments and feature ratings of contextually relevant concepts. Furthermore, we describe a novel, computationally tractable method for improving predictions of contextually-unconstrained embedding models based on dimensionality reduction of their internal representation to a small number of contextually relevant semantic features. By improving the correspondence between predictions derived automatically by machine learning methods using

show abstract

Section: Contextual Projection: Defining Feature Vectors In Embedding...mentioning

confidence: 99%

Section: Experiments 2: Contextual Projection Captures Reliable Infor...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large‐Scale Text Corpora

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The technique of averaging sentence fMRI is commonly used in imaging studies for that reason (Anderson et al, 2016 ; Just et al, 2017 ; Grand et al, 2018 ). In this case it is specifically supported by neurological evidence suggesting that sentence comprehension consist of a core representation of several word meanings encoded across the brain (Gennari et al, 2007 ; Anderson et al, 2016 ).…”

Section: Data Collection and Processingmentioning

confidence: 99%

How the Brain Dynamically Constructs Sentence-Level Meanings From Word-Level Features

Aguirre-Celis¹,

Miikkulainen²

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

How are words connected to the thoughts they help to express? Recent brain imaging studies suggest that word representations are embodied in different neural systems through which the words are experienced. Building on this idea, embodied approaches such as the Concept Attribute Representations (CAR) theory represents concepts as a set of semantic features (attributes) mapped to different brain systems. An intriguing challenge to this theory is that people weigh concept attributes differently based on context, i.e., they construct meaning dynamically according to the combination of concepts that occur in the sentence. This research addresses this challenge through the Context-dEpendent meaning REpresentations in the BRAin (CEREBRA) neural network model. Based on changes in the brain images, CEREBRA quantifies the effect of sentence context on word meanings. Computational experiments demonstrated that words in different contexts have different representations, the changes observed in the concept attributes reveal unique conceptual combinations, and that the new representations are more similar to the other words in the sentence than to the original representations. Behavioral analysis further confirmed that the changes produced by CEREBRA are actionable knowledge that can be used to predict human responses. These experiments constitute a comprehensive evaluation of CEREBRA's context-based representations, showing that CARs can be dynamic and change based on context. Thus, CEREBRA is a useful tool for understanding how word meanings are represented in the brain, providing a framework for future interdisciplinary research on the mental lexicon.

show abstract

Reduced social attention in autism is magnified by perceptual load in naturalistic environments

et al. 2022

View full text Add to dashboard Cite

Individuals with autism spectrum conditions (ASC) describe differences in both social cognition and sensory processing, but little is known about the causal relationship between these disparate functional domains. In the present study, we sought to understand how a core characteristic of autism-reduced social attention-is impacted by the complex multisensory signals present in real-world environments. We tested the hypothesis that reductions in social attention associated with autism would be magnified by increasing perceptual load (e.g., motion, multisensory cues). Adult participants (N = 40; 19 ASC) explored a diverse set of 360 real-world scenes in a naturalistic, active viewing paradigm (immersive virtual reality + eyetracking). Across three conditions, we systematically varied perceptual load while holding the social and semantic information present in each scene constant. We demonstrate that reduced social attention is not a static signature of the autistic phenotype. Rather, group differences in social attention emerged with increasing perceptual load in naturalistic environments, and the susceptibility of social attention to perceptual load predicted continuous measures of autistic traits across groups. Crucially, this pattern was specific to the social domain: we did not observe differential impacts of perceptual load on attention directed toward nonsocial semantic (i.e., object, place) information or low-level fixation behavior (i.e., overall fixation frequency or duration). This study provides a direct link between social and sensory processing in autism. Moreover, reduced social attention may be an inaccurate characterization of autism. Instead, our results suggest that social attention in autism is better explained by "social vulnerability," particularly to the perceptual load of real-world environments. Lay SummaryReal-world sensory contexts place incredible demands on selective attention. Autistic individuals report a particular challenge navigating contexts with high perceptual load, but little is known about its impact on classic, social signatures of autism. Here, we used eyetracking in immersive virtual reality to test the impact of perceptual load on social attention in naturalistic contexts. Overall, we found that group differences in social attention emerge with increasing real-world perceptual load, revealing a novel link between social and sensory processing in autism.

show abstract

Semantic projection recovers rich human knowledge of multiple object features from word embeddings

Cited by 65 publications

References 81 publications

Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large‐Scale Text Corpora

Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large‐Scale Text Corpora

How the Brain Dynamically Constructs Sentence-Level Meanings From Word-Level Features

Reduced social attention in autism is magnified by perceptual load in naturalistic environments

Contact Info

Product

Resources

About