Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions

Abstract

Models that represent meaning as high-dimensional numerical vectors—such as latent semantic analysis (LSA), hyperspace analogue to language (HAL), bound encoding of the aggregate language environment (BEAGLE), topic models, global vectors (GloVe), and word2vec—have been introduced as extremely powerful machine-learning proxies for human semantic representations and have seen an explosive rise in popularity over the past 2 decades. However, despite their considerable advancements and spread in the cognitive sciences, one can observe problems associated with the adequate presentation and understanding of some of their features. Indeed, when these models are examined from a cognitive perspective, a number of unfounded arguments tend to appear in the psychological literature. In this article, we review the most common of these arguments and discuss (a) what exactly these models represent at the implementational level and their plausibility as a cognitive theory, (b) how they deal with various aspects of meaning such as polysemy or compositionality, and (c) how they relate to the debate on embodied and grounded cognition. We identify common misconceptions that arise as a result of incomplete descriptions, outdated arguments, and unclear distinctions between theory and implementation of the models. We clarify and amend these points to provide a theoretical basis for future research and discussions on vector models of semantic representation.

Machine learning as a model for cultural learning: Teaching an algorithm what it means to be fat

“…Research in distributed representations, when combined with an approach to machine-learning called artificial neural networks (ANNs), has yielded a strategy to model the meaning encoded in language, called word2vec. Word2vec also stands out from other computational methods to analyze text for its performance on linguistic tasks, such as recognizing analogies and synonyms, and because it mirrors aspects of human cognition (Günther et al, 2019).…”

“…Since word2vec with CBOW learns word-vectors by predicting words given their contexts, words that occur in similar contexts are represented with similar word-vectors. 8 Word2vec is theoretically motivated by the linguistic 5 Equivalently, to the activation of the hidden layer triggered by a particular words, (Günther et al, 2019). 6 In a word2vec model, the N specific basis vectors ("axes") picked out by the learning process are arbitrary, but the word embeddings encode a (latent) coordinate system in which dimensions correspond to salient dimensions of language.…”

“…7 As we shall see, the learning algorithm is also cognitively plausible (Günther et al, 2019). 8 It is important to note that word2vec is not merely learning word co-occurence.…”

“…This means that word2vec is sensitive not to contiguity, i.e., raw co-occurence, but rather to contingency, i.e., the degree to which b is predictive of a (Günther et al, 2019). This turns out to make word2vec more realistic at an algorithmic and hardware level (Günther et al, 2019) 9 Mathematically, each context word is represented in the input layer as a vector with V elements, one for each word in the vocabulary V, so |V| = V . Each vector is "one-hot" encoded, meaning it is a vector of 0's and a single 1; the element corresponding to the context word is indicated by a 1, and the elements corresponding to all other vocabulary words are 0.…”

“…Word2vec uses an artificial neural network (ANN) to learn semantically rich representations of words. Its representational scheme and algorithmic structure are inspired (in broad strokes) by the architecture of the human brain and its learning processes (Miller et al, 1995;Mitchell et al, 2008), while the overall computational strategy emerges from a long tradition tradition in cognitive science (Günther et al, 2019) that models semantic memory as high-dimensional vectors learned from the way meaningful units are jointly distributed across experience (Jones et al, 2015). As we will show, word2vec allows us to model the learning of schemata at multiple levels and at large-scale-while responding to older calls to integrate cultural sociology with cognition (DiMaggio, 1997;Strauss and Quinn, 1997) and more recent calls to integrate culture and cognition research with computational thinking (Foster, 2018).…”