Learning linear transformations between counting-based and prediction-based word embeddings

Abstract

Despite the growing interest in prediction-based word embedding learning methods, it remains unclear as to how the vector spaces learnt by the prediction-based methods differ from that of the counting-based methods, or whether one can be transformed into the other. To study the relationship between counting-based and prediction-based embeddings, we propose a method for learning a linear transformation between two given sets of word embeddings. Our proposal contributes to the word embedding learning research in three ways: (a) we propose an efficient method to learn a linear transformation between two sets of word embeddings, (b) using the transformation learnt in (a), we empirically show that it is possible to predict distributed word embeddings for novel unseen words, and (c) empirically it is possible to linearly transform counting-based embeddings to prediction-based embeddings, for frequent words, different POS categories, and varying degrees of ambiguities.

Closed Form Word Embedding Alignment

“…A couple of very recent papers propose methods to align embeddings after their construction, but focus on affine transformations, as opposed to the more restrictive but distance preserving rotations of our method. Bollegala et al [4] uses gradient descent, for parameter γ, to directly optimize…”

“…GloVe: [4] The GloVe model is a log-bilinear model based on ratios of word-word co-occurrence frequencies. The training objective is for the dot product of the vectors learned for words to equal the logarithm of their cooccurrence frequency.…”

“…Given some textual data, there are several ways of extracting a distributed vector representation or an embedding for the words in the data. Our default approach will be GloVe [4], which is prediction based. We also consider word2vec [23,21], the other most common prediction-based approach, using the Gensim [29] implementation.…”

Restricted Isometry Property under High Correlations

“…Linear transformations are regularly used to transfer between different embeddings or to adapt to a new domain (Bollegala et al, 2017;Arora et al, 2018b). The linear transformation can encode contextual information, an idea utilized recently by (Khodak et al, 2018) who applied a linear transformation on the DisC embedding scheme to construct a new embedding scheme (referred to as à la carte embedding), and empirically showed that it outperforms many other popular word sequence embedding schemes.…”

Refining Pretrained Word Embeddings Using Layer-wise Relevance Propagation

“…One promising approach is the use of other information such as multimodal information (Bruni et al, 2014;Kiela et al, 2014;Kiela and Clark, 2015;Kiela et al, 2015a;Silberer et al, 2017) and language resources Kiela et al, 2015b;Rothe and Schütze, 2017;Yu and Dredze, 2014). Other refinement methods include task-specific embeddings (Bolukbasi et al, 2016;Yu et al, 2017) and the selective use of multiple embeddings (Bollegala et al, 2017;Kiela et al, 2018).…”