Indirect associations in learning semantic and syntactic lexical relationships

Abstract

Computational models of distributional semantics (a.k.a. word embeddings) represent a word's meaning in terms of its relationships with all other words. We examine what grammatical information is encoded in distributional models and investigate the role of indirect associations. Distributional models are sensitive to associations between words at one degree of separation, such as 'tiger' and 'stripes', or two degrees of separation, such as 'soar' and 'fly'. By recursively adding higher levels of representations to a computational, holographic model of semantic memory, we construct a distributional model sensitive to associations between words at arbitrary degrees of separation. We find that word associations at four degrees of separation increase the similarity assigned by the model to English words that share part-of-speech or syntactic type. Word associations at four degrees of separation also improve the ability of the model to construct grammatical English sentences. Our model proposes that human memory uses indirect associations to learn part-of-speech and that the basic associative mechanisms of memory and learning support knowledge of both semantics and grammatical structure.

Maze Learning using a Hyperdimensional Predictive Processing Cognitive Architecture

“…We choose MINERVA 2 since it captures a wide variety of human memory phenomena across differing settings and, as such, seems a good candidate for integration into a cognitive architecture. MINERVA 2 has been applied to a variety of experimental paradigms, including judgement of frequency and recognition [15], category learning [16], implicit learning [18,19], associative and reinforcement learning phenomena from both the animal and human learning literature [7,17], heuristics and biases in decision-making [8], hypothesis-generation [50], learning the meaning of words [20,31], and the production of grammatical sentences [22,24].…”

“…The continuous growth of the memory table results in a scaling problem for CogNGen, with significant slow downs even in the small maze learning tasks under consideration in this paper. Most MINERVA 2 models store only a small number of memory traces, though a few MINERVA 2 models used for language processing have stored up to 20,000 [20] to 500,000 traces [21,24]. With a persistent long-term memory store across learning the maze task, in the worst case, as many as millions of traces might be stored in CogNGen's memory.…”

Towards a Predictive Processing Implementation of the Common Model of Cognition

“…Each holographic vector represents a distinct concept, collectively serving as the basis vectors for the agent's conceptual space [20]. Our model is able accounts for human performance on a wide range of tasks, including recall, probability judgement, and decision-making [13], as well as how humans learn the meaning and part-of-speech of words from experience [14].…”

Holographic Declarative Memory: Distributional semantics as the architecture of memory

“…2 The question-based encoding used by HDM allows the model to be structured around the atomic items of experience-values or concepts-rather than the experiences-chunks or episodes-themselves. The encoding technique used by HDM has proven effective as a method of modelling the semantic (Jones & Mewhort, 2007) and syntactic (Kelly, Ghafurian, West, & Reitter, 2020) knowledge stored in the mental lexion, but here we explore its utility as a general purpose scheme for declarative memory.…”

“…Model, a recursive variant of BEAGLE with multiple levels of representations, is able to learn arbitrarily abstract relationships (Kelly et al, 2020). Sensitivity to abstract relationships is useful for capturing sytactic similarity between words, for ordering words into grammatical sentences, and for being able to distinguish between grammatical and ungrammatical word orderings, even in the case of nonsensical sentences that lack semantics.…”