Context-Dependent Sense Embedding

Abstract

Word embedding has been widely studied and proven helpful in solving many natural language processing tasks. However, the ambiguity of natural language is always a problem on learning high quality word embeddings. A possible solution is sense embedding which trains embedding for each sense of words instead of each word. Some recent work on sense embedding uses context clustering methods to determine the senses of words, which is heuristic in nature. Other work creates a probabilistic model and performs word sense disambiguation and sense embedding iteratively. However, most of the previous work has the problems of learning sense embeddings based on imperfect word embeddings as well as ignoring the dependency between sense choices of neighboring words. In this paper, we propose a novel probabilistic model for sense embedding that is not based on problematic word embedding of polysemous words and takes into account the dependency between sense choices. Based on our model, we derive a dynamic programming inference algorithm and an ExpectationMaximization style unsupervised learning algorithm. The empirical studies show that our model outperforms the state-of-the-art model on a word sense induction task by a 13% relative gain.

Dynamic Meta-Embeddings for Improved Sentence Representations

“…There has also been work on learning multiple embeddings per word Neelakantan et al, 2015;Vu and Parker, 2016), including a lot of work in sense embeddings where the senses of a word have their own individual embeddings (Iacobacci et al, 2015;Qiu et al, 2016), as well as on how to apply such sense embeddings in downstream NLP tasks (Pilehvar et al, 2017).…”

From Word To Sense Embeddings: A Survey on Vector Representations of Meaning

“…Ideally, a model should model the dependency between sense choices in order to address the ambiguity from context words. Qiu et al (2016) addressed this problem by proposing a pure sense-based model. The model also expands the disambiguation context from a small window (as done in the previous works) to the whole sentence.…”

MUSE: Modularizing Unsupervised Sense Embeddings

“…Recently, Qiu et al (2016) proposed an EM algorithm to learn purely sense-level representations, where the computational cost is high when decoding the sense identity sequence, because it takes exponential time to search all sense combination within a context window. Our modular design addresses such drawback, where the sense selection module decodes a sense sequence with linear-time complexity, while the sense representation module remains representation learning in the pure sense level.…”

“…The sense selection module decides which sense to use given a text context, whereas the sense representation module learns meaningful representations based on its statistical characteristics. Unlike prior work that must suffer from either inefficient sense selection (Qiu et al, 2016) or coarse-grained representation learning (Neelakantan et al, 2014;Li and Jurafsky, 2015;Bartunov et al, 2016), the proposed modularized framework is capable of performing efficient sense selection and learning representations in pure sense level simultaneously.…”

“…Under this framework, prior work focused on designing a single model to deliver both mechanisms (Neelakantan et al, 2014;Li and Jurafsky, 2015;Qiu et al, 2016). However, the previously proposed models introduce side-effects: 1) mixing word-level and sense-level tokens achieves efficient sense selection but introduces ambiguous word-level tokens during the representation learning process (Neelakantan et al, 2014;Li and Jurafsky, 2015), and 2) pure sense-level tokens prevent ambiguity from word-level tokens but require exponential time complexity when decoding a sense sequence (Qiu et al, 2016).…”