Inductive Entity Representations from Text via Link Prediction

Abstract

Knowledge Graphs (KG) are of vital importance for multiple applications on the web, including information retrieval, recommender systems, and metadata annotation.Regardless of whether they are built manually by domain experts or with automatic pipelines, KGs are often incomplete. To address this problem, there is a large amount of work that proposes using machine learning to complete these graphs by predicting new links. Recent work has begun to explore the use of textual descriptions available in knowledge graphs to learn vector representations of entities in order to preform link prediction. However, the extent to which these representations learned for link prediction generalize to other tasks is unclear. This is important given the cost of learning such representations. Ideally, we would prefer representations that do not need to be trained again when transferring to a different task, while retaining reasonable performance.Therefore, in this work, we propose a holistic evaluation protocol for entity representations learned via a link prediction objective. We consider the inductive link prediction and entity classification tasks, which involve entities not seen during training. We also consider an information retrieval task for entity-oriented search. We evaluate an architecture based on a pretrained language model, that exhibits strong generalization to entities not observed during training, and outperforms related state-of-the-art methods (22% MRR improvement in link prediction on average). We further provide evidence that the learned representations transfer well to other tasks without fine-tuning. In the entity classification task we obtain an average improvement of 16% in accuracy compared with baselines that also employ pre-trained models. In the information retrieval task, we obtain significant improvements of up to 8.8% in NDCG@10 for natural language queries. We thus show that the learned representations are not limited KG-specific tasks, and have greater generalization properties than evaluated in previous work. CCS CONCEPTS• Computing methodologies → Statistical relational learning; • Information systems → Information retrieval.This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

Relational World Knowledge Representation in Contextual Language Models: A Review

“…Pooling entity representations (Baldini Soares et al, 2019;) Embedding relations externally (Wang et al, 2021d;Daza et al, 2021) Treating relations as tokens (Bosselut et al, 2019;Hwang et al, 2021) English Wikipedia has around 6 million entities. This can make pretraining on a larger vocabulary expensive in terms of both time and memory usage (Yamada et al, 2020;.…”

“…Non-contextual relation embeddings may be learned by defining a separate relation embedding matrix with |R| rows and fusing this matrix into the LM. One advantage of this approach, similar to methods for retrieving external entity embeddings ( § 4.3), is that it supports fusion at both the late (Wang et al, 2021d;Daza et al, 2021) and middle (Liu et al, 2021c) stages. As an example of the former, Wang et al (2021d) propose an LM pretraining objective whereby textual descriptions of KB entities are input to and encoded by an LM, then combined with externally-learned relation embeddings at the output using a link prediction loss (Figure 5b).…”

Knowledge graph embedding for data mining vs. knowledge graph embedding for link prediction – two sides of the same coin?

“…While most of those approaches only consider graphs with nodes and edges, most knowledge graphs also contain literals, e.g., strings and numeric values. Recently, approaches combining textual information with knowledge graph embeddings using language modeling techniques have also been proposed, using techniques such as word2vec and convolutional neural networks [45] or transformer methods [9,43]. [11] shows a survey of approaches which take such literal information into account.…”

“…Portisch et al / Knowledge graph embedding for data mining vs. knowledge graph embedding for link prediction 13h, r, e 1 and h, r, e 2 ,9 or, less strongly, if there exists a chain of such statements. More formally, we can write the notion of similarity between two entities in link prediction approaches as e 1 ≈ e 2 ← ∃t, r : r(e 1 , t) ∧ r(e 2 , t)(19) e 1 ≈ e 2 ← ∃h, r : r(h, e 1 ) ∧ r(h, e 2 ) (…”

PGE: Robust Product Graph Embedding Learning for Error Detection

“…For example, KEPLER [35] proposes to encode textual entity descriptions with BERT as their embeddings, and then jointly optimize the KGE and language modeling objectives. BLP [13] trains PLM and KG in an end-to-end manner. Since the language modeling objective of PLM suffer from high computational cost and require a large corpus for training, it is time consuming to apply these methods to large scale KGs.…”