Embedding Words and Senses Together via Joint Knowledge-Enhanced Training

Abstract

Word embeddings are widely used in Natural Language Processing, mainly due to their success in capturing semantic information from massive corpora. However, their creation process does not allow the different meanings of a word to be automatically separated, as it conflates them into a single vector. We address this issue by proposing a new model which learns word and sense embeddings jointly. Our model exploits large corpora and knowledge from semantic networks in order to produce a unified vector space of word and sense embeddings. We evaluate the main features of our approach both qualitatively and quantitatively in a variety of tasks, highlighting the advantages of the proposed method in comparison to stateof-the-art word-and sense-based models.

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval

“…However, authors in [9] have shown that traditional word embeddings are not able to cope with the polysemy problem. Recently, some work [3,14] have tackled this issue. Cheng et al [3] propose to extend the skip-gram model [15] to identify the relevant word-concept pairwise given a context by jointly training the corresponding embeddings.…”

“…The connection between words and concepts is set up based on either implicit senses (corpus-based) or explicit senses (inventoried in a knowledge resource). In the same spirit, Mancini et al [14] focus on the separate representation of each word sense according to only discrete senses provided by a semantic network (WordNet). A unified extended neural CBOW architecture allows building a shared vector space of both words and senses.…”

Semantic similarity analysis and application in knowledge graphs

“…Moreover, we also use a word overlap based approach (Lesk, 1986) for comparison. Finally, as we are actually testing the word and synset inter-connectivity in a shared vector space, we also include results from a similar experiment from (Mancini et al, 2016), and denote them as SW2V (Mancini et al, 2016) and AutoExtend (Rothe and Schütze, 2015). Apart from the four Synset2Vec models, we also use a combination of WSD3, M4, and a back-off strategy (Camacho-Collados et al, 2016) (M4+WNFS) for comparing to SW2V, since such strategy is also included.…”

“…These WSD systems are not dependent on general sense or word representation, but only rely on similarity metrics and underlying semantic resources. Recent works have tried to learn distinct representations for individual word senses based on clustering approaches (Reisinger and Mooney, 2010;Huang et al, 2012), mixed or hybrid sense and word embedding (Chen et al, 2014;Iacobacci et al, 2015), new embedding architectures (Mancini et al, 2016) and various knowledge sources (Rothe and Schütze, 2015). Embedding-based sense and word representation models have shown promising performance in WSD tasks , and they have better general-ization power of the vector representation of words, compared to similarity-based approaches.…”

“…SensEmbed (Iacobacci et al, 2015) uses BabelNet (Navigli and Ponzetto, 2012) as sense catalog and relies on CBOW model to learn sense embeddings from automatically sense-annotated corpora, where senses are discriminated with a WSD system, Babelfy (Moro et al, 2014). A consequent SW2V model (Mancini et al, 2016) extends the input and output layer of the neural network architecture with word senses in CBOW model, so that word and sense embeddings can be trained jointly exploiting knowledge from both text corpora and large semantic networks. Apart from representing senses and words in the same semantic vector space, Nasari presented joint representation of senses, words and named entities.…”