Tailoring Continuous Word Representations for Dependency Parsing

Abstract

Word representations have proven useful for many NLP tasks, e.g., Brown clusters as features in dependency parsing (Koo et al., 2008). In this paper, we investigate the use of continuous word representations as features for dependency parsing. We compare several popular embeddings to Brown clusters, via multiple types of features, in both news and web domains. We find that all embeddings yield significant parsing gains, including some recent ones that can be trained in a fraction of the time of others. Explicitly tailoring the representations for the task leads to further improvements. Moreover, an ensemble of all representations achieves the best results, suggesting their complementarity.

Not All Contexts Are Created Equal: Better Word Representations with Variable Attention

“…For syntactic tasks, it has been shown that increasing the window size can adversely impact in the quality of the embeddings (Bansal et al, 2014;Lin et al, 2015).…”

“…While these models are adequate for learning semantic features, one of the problems of this model is the lack of sensitivity for word order, which limits their ability of learn syntactically motivated embeddings (Ling et al, 2015a;Bansal et al, 2014). While models have been proposed to address this problem, the complexity of these models ("Structured skip-n-gram" and "CWindow") grows linearly as size of the window of words considered increases, as a new set of parameters is created for each relative position.…”

Counter-fitting Word Vectors to Linguistic Constraints

“…This hypothesis supports unsupervised learning of meaningful word representations from large corpora (Curran, 2003;Ó Séaghdha and Korhonen, 2014;Mikolov et al, 2013;Pennington et al, 2014). Word vectors trained using these methods have proven useful for many downstream tasks including machine translation (Zou et al, 2013) and dependency parsing (Bansal et al, 2014).…”

Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words

“…word2vec and GloVe representations have provided state-of-the-art results on various word similarity and analogy detection task (Mikolov et al, 2013c;Mikolov et al, 2013b;Pennington et al, 2014). Word embedding based models are also used for other NLP tasks such as dependency parsing, semantic role labeling, POS tagging, NER, question-answering (Bansal et al, 2014;Collobert et al, 2011;Weston et al, 2015) and our work on LSSD is a novel application of word embeddings.…”