Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction

Abstract

We present a novel word level vector representation based on symmetric patterns (SPs). For this aim we automatically acquire SPs (e.g., "X and Y") from a large corpus of plain text, and generate vectors where each coordinate represents the cooccurrence in SPs of the represented word with another word of the vocabulary. Our representation has three advantages over existing alternatives: First, being based on symmetric word relationships, it is highly suitable for word similarity prediction. Particularly, on the SimLex999 word similarity dataset, our model achieves a Spearman's ρ score of 0.517, compared to 0.462 of the state-of-the-art word2vec model. Interestingly, our model performs exceptionally well on verbs, outperforming stateof-the-art baselines by 20.2-41.5%. Second, pattern features can be adapted to the needs of a target NLP application. For example, we show that we can easily control whether the embeddings derived from SPs deem antonym pairs (e.g. (big,small)) as similar or dissimilar, an important distinction for tasks such as word classification and sentiment analysis. Finally, we show that a simple combination of the word similarity scores generated by our method and by word2vec results in a superior predictive power over that of each individual model, scoring as high as 0.563 in Spearman's ρ on SimLex999. This emphasizes the differences between the signals captured by each of the models.

Is "Universal Syntax" Universally Useful for Learning Distributed Word Representations?

“…2(a) and e.g. (Schwartz et al, 2015)), verbs are apparently easier to model in Italian (Fig. 2(c) Table 3: Rankings based on Acc 1 scores over syntactic analogy groups (from the Google dataset).…”

Contradiction Detection with Contradiction-Specific Word Embedding

“…With the aim to resolve the sentiment contrast, a sentiment-specific word embedding [54] is learnt by weakly-supervised tweets collected by positive and negative emotions. Some neural network based models are proposed to revisit word embedding for lexical contrast [7][8][9][10]. There are two ways to get the contrasting pairs for learning embeddings.…”

“…Chen et al [7] and Mrksic et al [8] both use lexical resources to get antonym pairs; however, this is the small part of contrasting pairs. Schwartz et al [9] apply patterns to get contrasting pairs from web text such as a Wikipedia page. This method also meet the low coverage problem, because the number of the contrasting pairs which can be described by "from X to Y" or "either X or Y" is limited.…”

“…The appropriate mapping of contrasting words will lead to a high discrimination of contradictory sentences, as shown in Figure 2b. To address the issue, there are some recent methods [7][8][9][10] currently being investigated. Mrksic et al [8] apply lexical knowledge such as WordNet [11] and PPDB [12] (Paraphrase Database) (http://www.cis.upenn.edu/~ccb/ppdb/) to revise word embedding by using 12,802 antonym pairs and 31,828 synonymy pairs.…”

“…In addition, antonym pairs are just part of contrasting word pairs, and many other contrasting word pairs such as "shelve" and "pass" can not be obtained from the lexical resources. Schwartz et al [9] use patterns such as "from X to Y" and "either X or Y" to extract antonyms from Wikipedia. However, the pattern based methods would meet the data sparsity problem.…”