Commonsense Knowledge Base Completion with Structural and Semantic Context

Abstract

Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and ConceptNet) poses unique challenges compared to the much studied conventional knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form text to represent nodes, resulting in orders of magnitude more nodes compared to conventional KBs ( ∼18x more nodes in ATOMIC compared to Freebase (FB15K-237)). Importantly, this implies significantly sparser graph structures — a major challenge for existing KB completion methods that assume densely connected graphs over a relatively smaller set of nodes.In this paper, we present novel KB completion models that can address these challenges by exploiting the structural and semantic context of nodes. Specifically, we investigate two key ideas: (1) learning from local graph structure, using graph convolutional networks and automatic graph densification and (2) transfer learning from pre-trained language models to knowledge graphs for enhanced contextual representation of knowledge. We describe our method to incorporate information from both these sources in a joint model and provide the first empirical results for KB completion on ATOMIC and evaluation with ranking metrics on ConceptNet. Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1.5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency. Further analysis on model predictions shines light on the types of commonsense knowledge that language models capture well.

ConceptBert: Concept-Aware Representation for Visual Question Answering

“…In this work, we focus on the English vocabulary which contains approximately 1.5 million nodes. To avoid the step of the query construction and take full advantage of the large scale KG, we exploit ConceptNet embedding proposed in (Malaviya et al, 2020) and generate the KG representation k ∈ R n T ×d k .…”

“…Knowledge graph embedding: During our experiments, we explored different node embeddings for ConceptNet (e.g. GloVe (Pennington et al, 2014), NumberBatch (Speer et al, 2016), and (Malaviya et al, 2020)). We found that the embedding generated by (Malaviya et al, 2020) works best in our model.…”

DualTKB: A Dual Learning Bridge between Text and Knowledge Base

“…This allowed us to first explore whether the proposed method worked at all before evaluating a semi-supervised approach by creating a weak supervision from a mapping between ConceptNet triples and the original OMCS sentences. While KBs are often dense with short named entity descriptions for nodes, many nodes for commonsense KBs are parts of sentences, making them inherently sparse which impacts their performance as empirically studied by Malaviya et al (2020).…”

“…For path generation, we rely on a conventional KB completion task where the goal is to maximize the validity score of a tail entity e t given the pair (e h , r). For example, Malaviya et al (2020) addresses the challenges unique to commonsense KB completion due to sparsity and large numbers of nodes resulting from encoding commonsense facts. However, KB completion does not always equate generation of edges, with the exception of COMET from Bosselut et al (2019) that generates tail node e t given the pair (e h , r).…”

“…Since repurposing generative models for conventional KB completion evaluation is difficult (Malaviya et al, 2020;Bosselut et al, 2019), we propose a new commonsense KB completion evaluation task for generative models. It is close in spirit to conventional KB completion, but comes with its own set of challenges.…”