Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations

Abstract

Vector space representation of words has been widely used to capture fine-grained linguistic regularities, and proven to be successful in various natural language processing tasks in recent years. However, existing models for learning word representations focus on either syntagmatic or paradigmatic relations alone. In this paper, we argue that it is beneficial to jointly modeling both relations so that we can not only encode different types of linguistic properties in a unified way, but also boost the representation learning due to the mutual enhancement between these two types of relations. We propose two novel distributional models for word representation using both syntagmatic and paradigmatic relations via a joint training objective. The proposed models are trained on a public Wikipedia corpus, and the learned representations are evaluated on word analogy and word similarity tasks. The results demonstrate that the proposed models can perform significantly better than all the state-of-the-art baseline methods on both tasks.

The Word Analogy Testing Caveat

“…Results of the word analogy test are often accompanied by a visualisation of projected word vectors to the two dimensional plane using Principal Component Analysis (PCA) ( (Mikolov et al, 2013a;Sun et al, 2015) for example). Though these are generally not claimed to be part of the evaluation, the visualisations are included to convince the reader of the quality of the word embeddings with respect to word analogies-the line connecting a 1 and b 1 being approximately parallel to the line through a 2 and b 2 whenever word analogy recovery is optimal (as in Equation ( 2)).…”

Commonsense mining as knowledge base completion? A study on the impact of novelty

“…To approximate novelty, we use word embeddings (computed over the OMCS corpus) to calculate distance d(a, b) = ||head(a) − head(b)|| 2 + ||tail(a) − tail(b)|| 2 , where head and tail are X X X X X X X X represented by the average of word embeddings. Such a formulation is related to the concept of paradigmatic similarity (Sahlgren, 2006), and word embedding-based distance can approximate paradigmatic similarity (Sun et al, 2015). Two words are paradigmatically similar if one can be replaced for the other, while maintaining syntactical correctness of the sentence (e.g.…”

Embedding Compression with Isotropic Iterative Quantization

“…For some tasks, the evaluation is directly conducted over the embedding (e.g., measuring the cosine similarity between word vectors); whereas for others, a classifier is Pre-trained Embedding. We perform experiments with the GloVe embedding (Pennington, Socher, and Manning 2014) and the HDC embedding (Sun et al 2015). The GloVe embedding is trained from 42B tokens of Common Crawl data.…”