The mechanism of additive composition

Abstract

Additive composition (Foltz et al. in Discourse Process 15:285-307, 1998; Landauer and Dumais in Psychol Rev 104(2):211, 1997; Mitchell and Lapata in Cognit Sci 34(8):1388-1429, 2010) is a widely used method for computing meanings of phrases, which takes the average of vector representations of the constituent words. In this article, we prove an upper bound for the bias of additive composition, which is the first theoretical analysis on compositional frameworks from a machine learning point of view. The bound is written in terms of collocation strength; we prove that the more exclusively two successive words tend to occur together, the more accurate one can guarantee their additive composition as an approximation to the natural phrase vector. Our proof relies on properties of natural language data that are empirically verified, and can be theoretically derived from an assumption that the data is generated from a Hierarchical Pitman-Yor Process. The theory endorses additive composition as a reasonable operation for calculating meanings of phrases, and suggests ways to improve additive compositionality, including: transforming entries of distributional word vectors by a function that meets a specific condition, constructing a novel type of vector representations to make additive composition sensitive to word order, and utilizing singular value decomposition to train word vectors.

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

“…First, choose a word w. Then, for each window s containing w, take the average of the vectors of the words in s and denote it as v s . Now, take the average of v s for all the windows s containing w, and denote the average as u. Theorem 1 says that u can be mapped to the word vector v w by a linear transformation that does not depend on w. This linear structure may also have connections to some other phenomena related to linearity, e.g., Gittens et al (2017) and Tian et al (2017). Exploring such connections is left for future work.…”

Learning Semantically and Additively Compositional Distributional Representations

“…However, previous model designs mostly rely on linguistic intuitions (Paperno et al, 2014, inter alia), whereas our model has an exact logic interpretation. Furthermore, by using additive composition we enjoy a learning guarantee (Tian et al, 2015).…”

“…The rationale for our model is as follows. First, recent research has shown that additive composition of word vectors is an approximation to the situation where two words have overlapping context (Tian et al, 2015); therefore, it is suitable to implement an "and" or intersection operation (Section 3). We design our model such that the resulted distributional representations are expected to have additive compositionality.…”

Unsupervised sentence representations as word information series: Revisiting TF–IDF

“…This inequality reflects the difference between the importance of the constituents (i.e. the word embeddings Tian et al (2017), coefficients α, β are scalars drawn from a monotonic function. In this work, we consider that a reasonable choice for such a monotonic function is the Shannon's entropy (Shannon, 1949;Charniak, 1996;Aizawa, 2003).…”