Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large‐Scale Text Corpora

Abstract

Applying machine learning algorithms to automatically infer relationships between concepts from large-scale collections of documents presents a unique opportunity to investigate at scale how human semantic knowledge is organized, how people use it to make fundamental judgments ("How similar are cats and bears?"), and how these judgments depend on the features that describe concepts (e.g., size, furriness). However, efforts to date have exhibited a substantial discrepancy between algorithm predictions and human empirical judgments. Here, we introduce a novel approach to generating embeddings for this purpose motivated by the idea that semantic context plays a critical role in human judgment. We leverage this idea by constraining the topic or domain from which documents used for generating embeddings are drawn (e.g., referring to the natural world vs. transportation apparatus). Specifically, we trained state-of-the-art machine learning algorithms using contextually-constrained text corpora (domain-specific subsets of Wikipedia articles, 50+ million words each) and showed that this procedure greatly improved predictions of empirical similarity judgments and feature ratings of contextually relevant concepts. Furthermore, we describe a novel, computationally tractable method for improving predictions of contextually-unconstrained embedding models based on dimensionality reduction of their internal representation to a small number of contextually relevant semantic features. By improving the correspondence between predictions derived automatically by machine learning methods using

Индивидуальные Различия В Ассоциативном Значении Слова Сквозь Призму Языковой Модели И Семантического Дифференциала

“…T. Merten отмечает положительную корреляцию между выраженностью психотизма и числом уникальных ассоциативных реакций в различных условиях проведения ассоциативного эксперимента (Merten, 1993). В названных работах, а также в исследовании (Innes, 1972) (Черкасова, 2008), содержащего 253 стимула, отобранных по материалам трех ассоциативных опросов, проведенных с интервалами 10-20 лет с носителями русского языка. В данный словарь вошли русские слова-стимулы, которые повторялись в трех или двух массовых ассоциативных опросах, по материалам которых опубли-кованы важнейшие ассоциативные словари русского языка.…”

THINGS-data: A multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

“…To identify the degree to which the updated embedding yielded improved prediction of fine-grained similarity, we used 8 existing datasets from three studies [49][50][51] that had examined within category similarity. Note that predicted similarities are likely underestimated, given that the original similarity datasets were collected using different image examples and/or tasks.…”

“…Third, while increases in dataset size did not lead to notable improvements in overall performance, did increasing the dataset size improve more fine-grained predictions of similarity? To address this question, we used several existing datasets of within-category similarity ratings [49][50][51] and computed similarity predictions. Rather than computing similarity across all possible triplets, these predictions were constrained to triplet contexts within superordinate categories (e.g.…”

Beyond the Benchmarks: Toward Human-Like Lexical Representations

“…Word embeddings are largely founded on the notion of semantic similarity, and ensuring that word vector similarities match human judgments has been an important goal (e.g., Baroni et al, 2014 ; Pereira et al, 2016 ; An et al, 2018 ; Grand et al, 2018 ; Iordan et al, 2022 ). Less attention has been paid to whether the actual structure of a DSM's similarity space matches what is known about the human lexicon.…”

“…asteroid, belt , and buckle , Griffiths et al, 2007 )—are not consistently captured in embedding spaces (Griffiths et al, 2007 ; Nematzadeh et al, 2017 ; Rodriguez and Merlo, 2020 ). Building on the insight from Griffiths et al ( 2007 ) that interpretation of a word within the context of a topic can resolve some of these mismatches with human judgments by appropriately disambiguating the words, one avenue for the future may be to consider word embeddings that are topically-constrained (such as in Iordan et al, 2022 ).…”