Composition in Distributional Models of Semantics

Abstract

Distributional models of semantics have proven themselves invaluable both in cognitive modelling of semantic phenomena and also in practical applications. For example, they have been used to model judgments of semantic similarity (McDonald, 2000) and association (Denhire and Lemaire, 2004;Griffiths et al., 2007) and have been shown to achieve human level performance on synonymy tests (Landuaer and Dumais, 1997;Griffiths et al., 2007) such as those included in the Test of English as Foreign Language (TOEFL). This ability has been put to practical use in automatic thesaurus extraction (Grefenstette, 1994). However, while there has been a considerable amount of research directed at the most effective ways of constructing representations for individual words, the representation of larger constructions, e.g., phrases and sentences, has received relatively little attention. In this thesis we examine this issue of how to compose meanings within distributional models of semantics to form representations of multi-word structures.Natural language data typically consists of such complex structures, rather than just individual isolated words. Thus, a model of composition, in which individual word meanings are combined into phrases and phrases combine to form sentences, is of central importance in modelling this data. Commonly, however, distributional representations are combined in terms of addition (Landuaer and Dumais, 1997;Foltz et al., 1998), without any empirical evaluation of alternative choices. Constructing effective distributional representations of phrases and sentences requires that we have both a theoretical foundation to direct the development of models of composition and also a means of empirically evaluating those models.The approach we take is to first consider the general properties of semantic composition and from that basis define a comprehensive framework in which to consider the composition of distributional representations. The framework subsumes existing proposals, such as addition and tensor products, but also allows us to define novel composition functions. We then show that the effectiveness of these models can be i evaluated on three empirical tasks.The first of these tasks involves modelling similarity judgements for short phrases gathered in human experiments. Distributional representations of individual words are commonly evaluated on tasks based on their ability to model semantic similarity relations, e.g., synonymy or priming. Thus, it seems appropriate to evaluate phrase representations in a similar manner. We then apply compositional models to language modelling, demonstrating that the issue of composition has practical consequences, and also providing an evaluation based on large amounts of natural data. In our third task, we use these language models in an analysis of reading times from an eye-movement study. This allows us to investigate the relationship between the composition of distributional representations and the processes involved in comprehending phrases and sentences.We ...

Enhancing Sensitivity Classification with Semantic Features Using Word Embeddings

“…In that study, the authors obtained document vector representations by deploying simple composition functions (e.g. min, average, max) to construct vector representations of combinations of words, such as phrases or sentences, from term vector models [20]. They showed that these compositional document vectors could be effectively used as features to extend text classification and improve classification performance.…”

“…They showed that these compositional document vectors could be effectively used as features to extend text classification and improve classification performance. In this work, we follow the methodology of [19,20] and compose document representations from word embeddings in the task of sensitivity classification. However, differently from Balikas and Amini [19], we show how these document representations combined with text features can be effective for discovering latent sensitivities.…”

Conceptual Versus Referential Affordance in Concept Composition

“…Recently, researchers have begun to explore compositional distributional semantics, giving a distributional representation not only to words but also to phrases and even sentences (Mitchell and Lapata 2010;Coecke et al 2011;Socher et al 2012;Baroni et al 2014a;Pham 2016 among many others); the previous work we presented at the end of Sect. 3.2 falls into this line of research.…”

“…A very simple but stubbornly effective method is to simply add up the word vectors, as in Fig. 4 (Mitchell and Lapata 2010), but more sophisticated methods have been designed that sometimes yield better results (Baroni and Zamparelli 2010). Nothing we say in this paper depends on the chosen method for composition, hence we will simply use comp( ⃖⃖⃖⃖⃖ ⃗ , ⃖⃖⃖⃖⃖⃖⃖⃖⃖ ⃗) for the distributional representation of the phrase red moon obtained by applying a composition function to its constituent word vectors, ⃖⃖⃖⃖⃖ ⃗ and ⃖⃖⃖⃖⃖⃖⃖⃖⃖ ⃗ (we represent word vectors with an overhead arrow).…”

The Pragmatics of Political Messages in Twitter Communication

“…Co-occurrence statistics allow to quantitatively project some of a word's semantics grounded in users' categorization performance ( [18]). Collocations show the relative most frequent (sometimes stereotypical, implicit) social categories in communication, but the research must be complimented with concordance analysis for semantic complexity.…”