Efficient knowledge graph accuracy evaluation

Abstract

Estimation of the accuracy of a large-scale knowledge graph (KG) often requires humans to annotate samples from the graph. How to obtain statistically meaningful estimates for accuracy evaluation while keeping human annotation costs low is a problem critical to the development cycle of a KG and its practical applications. Surprisingly, this challenging problem has largely been ignored in prior research. To address the problem, this paper proposes an efficient sampling and evaluation framework, which aims to provide quality accuracy evaluation with strong statistical guarantee while minimizing human efforts. Motivated by the properties of the annotation cost function observed in practice, we propose the use of cluster sampling to reduce the overall cost. We further apply weighted and two-stage sampling as well as stratification for better sampling designs. We also extend our framework to enable efficient incremental evaluation on evolving KG, introducing two solutions based on stratified sampling and a weighted variant of reservoir sampling. Extensive experiments on real-world datasets demonstrate the effectiveness and efficiency of our proposed solution. Compared to baseline approaches, our best solutions can provide up to 60% cost reduction on static KG evaluation and up to 80% cost reduction on evolving KG evaluation, without loss of evaluation quality.

KnowText: Auto-generated Knowledge Graphs for custom domain applications

“…Traditional Knowledge Graphs (generated through pre-defined ontology schema and manual assignment and definition of relation types) can be characterized by evaluating a number of quality metrics (Syntactic validity, Semantic accuracy, Consistency, Conciseness and Completeness) [6].…”

Creating Cybersecurity Knowledge Graphs From Malware After Action Reports

“…Some of the related work that has been done in this area, concentrated on validating RDF triples, against supporting or contradicting knowledge found from various other sources [23]. Another work [10] describes the cost of knowledge graph evaluation. It talks about various sampling methods which can help us choose triples at random and verify the correctness of those triples.…”

Biomedical Discovery through the integrative Biomedical Knowledge Hub (iBKH)

“…Such procedure is time- and cost-consuming, if one wants to evaluate sufficient triplets to reach the statistic criteria. To address this, for example, Gao et al [17] proposed an iterative evaluation framework for KG accuracy evaluation. Specifically, inspired by the properties of the annotation cost function observed in practice, the authors developed a cluster sampling strategy with unequal probability theory.…”