Words are Vectors, Dependencies are Matrices: Learning Word Embeddings from Dependency Graphs

Abstract

Distributional Semantic Models (DSMs) construct vector representations of word meanings based on their contexts. Typically, the contexts of a word are defined as its closest neighbours, but they can also be retrieved from its syntactic dependency relations. In this work, we propose a new dependencybased DSM. The novelty of our model lies in associating an independent meaning representation, a matrix, with each dependency-label. This allows it to capture specifics of the relations between words and contexts, leading to good performance on both intrinsic and extrinsic evaluation tasks. In addition to that, our model has an inherent ability to represent dependency chains as products of matrices which provides a straightforward way of handling further contexts of a word.

Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in Distributional Semantic Models

“…For both context-based and hybrid few-shot learning, we have achieved a new state of the art on 4 out of the 6 evaluation tasks used, showing that a careful, optimised approach can be the key to success in few-shot learning. Future work could explore other distributional models, such as dependency embeddings Czarnowska et al, 2019), but it is clear from our results that careful optimisation will be required to adapt other models to the few-shot setting.…”

Representing Syntax and Composition with Geometric Transformations

“…In both domains, the shared goals are: i) map entities v 2 V to embeddings e v where e 2 R |V|⇥n , n being the dimensionality of the vectors; ii) map relations r 2 R in one -or more -space R |R|⇥⇤ . In this work, we focus on constructing a syntactic dataset of positive training triples from a corpus as in Czarnowska et al (2019). All of the models we investigate rely on a negative sampling mechanism that generates a dataset D 0 of false triples.…”

“…Representing words in terms of their syntactic co-occurrences has been long proposed, both for count-based (Padó and Lapata, 2007;Weir et al, 2016), and neural (Hermann and Blunsom, 2013;Levy and Goldberg, 2014;Komninos and Manandhar, 2016;Czarnowska et al, 2019;Vashishth et al, 2019) models of word meaning. Tested on benchmark word similarity tasks, such models often perform favourably to models based on proximal co-occurrence, particularly when the similarity or substitutability of two words is considered rather than their relatedness (Levy and Goldberg, 2014).…”

“…There is of course an explosion in the number of parameters to be learnt in both DEP and EXT due to the many possible word-relation combinations which form the target vocabulary for these models (see Table 1). A possible solution, pro-posed by Czarnowska et al (2019), is the Dependency Matrix (DM) model which uses linear maps in the form of square matrices to encode relations. Here, the training objective is changed from predicting (target, context) pairs to (target, relation, context) triples, e.g., (rain,dobj,like).…”

Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

“…Vector addition (Rimell et al, 2016) .496 .472 Simplified Practical Lexical Function (Rimell et al, 2016) .496 .497 Vector addition (Czarnowska et al, 2019) .485 .475 Dependency vector addition (Czarnowska et al, 2019) .497 .439 Semantic functions (Emerson and Copestake, 2017b) .20 .16 Sem-func & vector ensemble (Emerson and Copestake, 2017b Previous work has shown that vector addition performs well on this task (Rimell et al, 2016;Czarnowska et al, 2019). I have trained a Skipgram model (Mikolov et al, 2013) using the Gensim library (Řehůřek and Sojka, 2010), tuning weighted addition on the dev set.…”