Random Walks and Neural Network Language Models on Knowledge Bases

Abstract

Random walks over large knowledge bases like WordNet have been successfully used in word similarity, relatedness and disambiguation tasks. Unfortunately, those algorithms are relatively slow for large repositories, with significant memory footprints. In this paper we present a novel algorithm which encodes the structure of a knowledge base in a continuous vector space, combining random walks and neural net language models in order to produce novel word representations. Evaluation in word relatedness and similarity datasets yields equal or better results than those of a random walk algorithm, using a dense representation (300 dimensions instead of 117K). Furthermore, the word representations are complementary to those of the random walk algorithm and to corpus-based continuous representations, improving the stateof-the-art in the similarity dataset. Our technique opens up exciting opportunities to combine distributional and knowledge-based word representations.

Embedding Senses for Efficient Graph-based Word Sense Disambiguation

“…Our assumption is that such a sequence can be considered a context of its starting node: a set of words that are related to, and can appear together in real texts with, the word sense represented by that node, thus emulating real text sentences; to what extent this assumption holds depends of course on the structure of the LKB we are using. Previous efforts in building word embeddings have shown the plausibility of this approach (Goikoetxea et al, 2015).…”

Towards Lexical Chains for Knowledge-Graph-basedWord Embeddings

“…As baselines, we evaluated two textcorpus-based word embeddings that are freely available on the web, as well as the best result of Goikoetxea et al (Goikoetxea et al, 2015), available from the UKB web page 9 . Thus, the pseudocorpus-based embeddings have been compared with text-based embeddings.…”

“…We reuse the sets of relations developed in these works to generate our Pseudo Corpora LC. Goikoetxea et al 2015(Goikoetxea et al, 2015 describe an architecture in which a run of the Random Walk algorithm (Agirre et al, 2014) produces an artificial corpus from WordNet. The graph that is fed to the algorithm is composed of WordNet synsets (the graph nodes) and of different types of relations between them (the graph arcs; some relation types are antonymy, hypernymy, derivation, etc.).…”

“…The corpora that the algorithms are trained on can contain either natural language text (e.g. Wikipedia or newswire articles) or artificiallygenerated pseudo corpora, such as the output of the Random Walk on Graphs algorithm, when run to select sequences of nodes from a knowledge graph (KG) -see (Goikoetxea et al, 2015) and (Ristoski and Paulheim, 2016). We denote the pseudo corpus generated via Random Walk on Graphs algorithm as Pseudo Corpus RWG.…”

Linked Disambiguated Distributional Semantic Networks

“…Complementary to this, a plethora of works in Natural Language Processing (NLP) has recently focused on combining knowledge bases with distributional information from text. These include approaches that modify Word2Vec [15] to learn sense embeddings [5], methods to enrich WordNet with embeddings for synsets and lexemes [21], acquire continuous word representations by combining random walks over knowledge bases and neural language models [11], or produce joint lexical and semantic vectors for sense representation from text and knowledge bases [4] In this paper, we follow this line of research and take it one step forward by producing a hybrid knowledge resource, which combines symbolic and statistical meaning representations while i) staying purely on the lexical-symbolic level, ii) explicitly distinguishing word senses, and iii) being human readable. Far from being technicalities, such properties are crucial to be able to embed a resource of this kind into the Semantic Web ecosystem, where human-readable distributional representations are explicitly linked to URIfied semantic resources.…”