Indirect associations in learning semantic and syntactic lexical relationships

Abstract

Computational models of distributional semantics represent word meanings in terms of words' relationships with all other words in a corpus. Although distributional models are sensitive to topic (e.g., tiger and stripes) and synonymy (e.g., soar and fly), the models have limited sensitivity to part-of-speech (e.g., book and shirt are nouns). How lexical-syntactic knowledge is encoded and how it meshes with semantic representations are open questions. Word co-occurrence relationships define a connected graph such that any two words have some degree of separation on the graph. Models of distributional semantics are typically sensitive to only one or two degrees of separation. By recursively adding higher levels of representations to a computational, holographic model of semantic memory, we build a model sensitive to arbitrary degrees of separation. We find that word associations at four degrees of separation increase the similarity between words that share part-of-speech or syntactic type and improve the ability of the model to construct grammatical sentences. Our model provides evidence that human memory must be sensitive to indirect associations to accommodate lexical syntactic relationships as well as evidence that semantics and syntax exist on a continuum that emerges from a unitary cognitive system.

Integrated, Not Isolated: Defining Typological Proximity in an Integrated Multilingual Architecture

“…Syntactic knowledge has rich stochastic ties to semantic representations, and can be seen as configural constraints that display a mix of regularities and exceptions. Distributed representations may well be a neurologically and psychologically plausible framework for syntactic knowledge, and it is a technically realistic candidate (Kelly et al, 2013 , 2017 ). At lexical, syntactic, and morphological levels, the overlap in semantic space and joint compressibility of lexicons associated with two languages determine their mutual facilitation.…”

A framework for computational models of human memory

“…The theoretical framework we propose allows for memory to be modelled at multiple levels of analysis, from the level of individual, biological neurons and messages passed between groups of those neurons (many-to-one models), to the level of the events processed by memory (one-to-one models), to the level of the concepts which emerge as aggregates across those events (many-to-many models), and on up, to the arbitrarily abstract concepts that emerge from aggregating across concepts (Kelly, Reitter, and West 2017). Furthermore, estimates of Bayesian probability arise from the vector algebra of many-to-many models (Kelly, Kwok, and West 2015).…”

“…Vectors representing items are typically randomly generated by the models, but ought to be generated according to semantic (Kelly, Reitter, and West 2017) or perceptual (Cox et al 2011;Kelly, Blostein, and Mewhort 2013) features, which would require integrating memory and perception.…”

Do We Need Neural Models to Explain Human Judgments of Acceptability?

“…In English, word order conveys much of the meaning of the sentence, and is critical in constructing a grammatical sentence. To account for this, we include in our analysis the Hierarchichal Holographic Model (HHM; Kelly et al, 2017), a model sensitive to the order of words in a sentence. HHM generates multiple levels of representations, such that higher levels are sensitive to more abstract relationships between words, such as partof-speech relationships (Kelly et al, 2017).…”

“…To account for this, we include in our analysis the Hierarchichal Holographic Model (HHM; Kelly et al, 2017), a model sensitive to the order of words in a sentence. HHM generates multiple levels of representations, such that higher levels are sensitive to more abstract relationships between words, such as partof-speech relationships (Kelly et al, 2017). We trained three levels of HHM representations with 1024 dimensions and a context window of 5 words to the left and right of each target word.…”