Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction

Abstract

Little is known about the trustworthiness of predictions made by knowledge graph embedding (KGE) models. In this paper we take initial steps toward this direction by investigating the calibration of KGE models, or the extent to which they output confidence scores that reflect the expected correctness of predicted knowledge graph triples. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false, and show that existing calibration techniques are effective for KGE under this common but narrow assumption. Next, we introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained. Here, we show that existing calibration techniques are much less effective under the OWA than the CWA, and provide explanations for this discrepancy. Finally, to motivate the utility of calibration for KGE from a practitioner's perspective, we conduct a unique case study of human-AI collaboration, showing that calibrated predictions can improve human performance in a knowledge graph completion task. * This work was done during an internship at Bloomberg.

How Certain is Your Transformer?

“…In future work, we are seeking to improve UEs quality obtained using the DPP dropout with the help of calibration (Safavi et al, 2020)…”

Approximate Knowledge Graph Query Answering: From Ranking to Binary Classification

“…In fact, it is often good practise to use so-called "hard" negative samples, which are similar to entities in T . A better alternative for finding entities not in T would be using more advanced techniques as proposed in [16].…”

A Hidden Challenge of Link Prediction: Which Pairs to Check?

“…We use Recall (R@k), the fraction of known missing/future links that are in the size-k set returned by the method, and Precision (P@k), the fraction of the k pairs that are known to be missing/future links. Recall is a more important metric, since (1) the returned set of pairs P does not contain final predictions, but rather pairs for a LP method to make final decisions about, and (2) our real-world graphs are inherently incomplete, and thus pairs returned that are not known to be missing links, could nonetheless be missing in the original dataset prior to ground-truth removal (i.e., the openworld assumption [25]). We report both in Table III.…”

“…Link prediction is a long-studied problem that attempts to predict either missing links in an incomplete graph, or links that are likely to form in the future. This has applications in discovering unknown protein interactions to speed up the discovery of new drugs, friend recommendation in social networks, knowledge graph completion, and more [1], [15], [16], [25]. Techniques range from heuristics, such as predicting links based on the number of common neighbors between a pair of nodes, to machine learning techniques, which formulate the link prediction problem as a binary classification problem over node pairs [7], [29].…”