A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors

Abstract

Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features. This paper introducesà la carte embedding, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings. Our method relies mainly on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. This transform is applicable "on the fly" in the future when a new text feature or rare word is encountered, even if only a single usage example is available. We introduce a new dataset showing how theà la carte method requires fewer examples of words in context to learn high-quality embeddings and we obtain state-of-the-art results on a nonce task and some unsupervised document classification tasks.

Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

“…where c = C∈C |C ∩ V| is the total number of words in C for which embeddings exist. In accordance with results reported by Khodak et al (2018), we found it helpful to apply a linear transformation to the so-obtained embedding, resulting in the final context embeddinĝ…”

“…Our approach is able to generate embeddings for OOV words even from only a single observation with high accuracy in many cases and outperforms previous work on the Definitional Nonce dataset (Herbelot and Baroni 2017) and the Contextual Rare Words dataset (Khodak et al 2018). To the best of our knowledge, this is the first work that jointly uses surface-form and context information to obtain representations for novel words.…”

“…In contrast to the sentences of the DN dataset, however, they are sampled randomly from the Westbury Wikipedia Corpus (WWC) (Shaoul and Westbury 2010) and, accordingly, do not have a definitional character in many cases. Khodak et al (2018) also provide a set of 300-dimensional word embeddings which, again, can be used to train our model. We may then compare the similarities of the so-obtained embeddings with the given similarity scores.…”

“…Importantly, both embeddings and their composition function are learned jointly, allowing each embedding to rely on its counterpart whenever its available information is not sufficient. In a similar fashion to work by Pinter, Guthrie, and Eisenstein (2017) and Khodak et al (2018), our approach is not trained from scratch, but instead makes use of preexisting word embeddings and aims to reconstruct these embeddings. This allows for a much faster learning process and enables us to easily combine our approach with any existing word embedding model, regardless of its internal structure.…”

Unsupervised Post-Processing of Word Vectors via Conceptor Negation

“…However, the method comes with a potential limitation: for each latent feature taking form as a PC of the word vectors, ABTT either completely removes the feature or keeps it intact. For this reason, Khodak et al (2018) argued that ABTT is liable either to not remove enough noise or to cause too much information loss. The objective of this paper is to address the limitations of ABTT.…”