CoDEx: A Comprehensive Knowledge Graph Completion Benchmark

Abstract

We present CODEX, a set of knowledge graph COmpletion Datasets EXtracted from Wikidata and Wikipedia that improve upon existing knowledge graph completion benchmarks in scope and level of difficulty. In terms of scope, CODEX comprises three knowledge graphs varying in size and structure, multilingual descriptions of entities and relations, and tens of thousands of hard negative triples that are plausible but verified to be false. To characterize CODEX, we contribute thorough empirical analyses and benchmarking experiments. First, we analyze each CODEX dataset in terms of logical relation patterns. Next, we report baseline link prediction and triple classification results on CODEX for five extensively tuned embedding models. Finally, we differentiate CODEX from the popular FB15K-237 knowledge graph completion dataset by showing that CODEX covers more diverse and interpretable content, and is a more difficult link prediction benchmark. Data, code, and pretrained models are available at https://bit.ly/2EPbrJs.

Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction

“…By contrast, ComplEx is trained with binary cross-entropy loss, the same loss that we use to calibrate models in the validation stage. (Kadlec et al, 2017;Safavi and Koutra, 2020). We conclude that for relation prediction under the CWA, vector scaling provides the best trade-off between calibration, accuracy, and efficiency, as it consistently improves accuracy and calibration with only O(k) extra parameters.…”

“…By contrast, at evaluation time we make an open-world assumption. Higher-quality validation negatives may alleviate this problem; indeed, recent works have raised this issue and constructed new datasets toward this direction, albeit for the task of triple classification (Pezeshkpour et al, 2020;Safavi and Koutra, 2020).…”

“…We are aware of only two recent works that investigate calibration for KGE, both of which address the task of triple classification (Tabacof and Costabello, 2020;Pezeshkpour et al, 2020). By contrast, we focus on link prediction, which is a different-and much more common (Safavi and Koutra, 2020)-KGE evaluation task. We also contribute an evaluation under the open-world assumption, whereas Tabacof and Costabello (2020) evaluate under the closed-world assumption only.…”

“…The link prediction task, which is most commonly used to evaluate KGE (Safavi and Koutra, 2020), is conducted as follows. Given a test triple (h, r, t), we hold out one of its entities or its relation to form a query (h, r, ?…”

“…Note that there have been recent (concurrent) efforts to construct appropriate datasets for evaluating KGE calibration (Pezeshkpour et al, 2020;Safavi and Koutra, 2020). Analysis on these new datasets is an important direction for future work.…”