Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations

Abstract

Learning good representations on multi-relational graphs is essential to knowledge base completion (KBC). In this paper, we propose a new self-supervised training objective for multirelational graph representation learning, via simply incorporating relation prediction into the commonly used 1vsAll objective. The new training objective contains not only terms for predicting the subject and object of a given triple, but also a term for predicting the relation type. We analyse how this new objective impacts multi-relational learning in KBC: experiments on a variety of datasets and models show that relation prediction can significantly improve entity ranking, the most widely used evaluation task for KBC, yielding a 6.1% increase in MRR and 9.9% increase in Hits@1 on FB15k-237 as well as a 3.1% increase in MRR and 3.4% in Hits@1 on Aristo-v4. Moreover, we observe that the proposed objective is especially effective on highly multi-relational datasets, i.e. datasets with a large number of predicates, and generates better representations when larger embedding sizes are used. Code will be available at https://github.com/facebookresearch/ssl-relation-prediction.

Multi-Task Learning and Improved TextRank for Knowledge Graph Completion

“…The PLM-based KGC models include LP-BERT [ 38 ], MTL-BERT [ 24 ], KG-XLnet [ 39 ], and KG-BERT [ 23 ]. Traditional KGC models include RESCAL-N3-RP [ 40 ], DensE [ 41 ], R-GCN [ 42 ], RotatE [ 20 ], ConvE [ 8 ], ComplEx [ 19 ], DistMult [ 18 ], and TransE [ 16 ]. In addition, we also used three description-based KGC models DKRL [ 21 ], ConMask [ 22 ], and OWE [ 9 ] as the traditional baseline models.…”

“…However, we found that the Hit@1 result was significantly worse than the translation distance models DensE [ 41 ] and RESCAL-N3-RP [ 40 ] on datasets WN18RR and FB15k-237. At the same time, regarding the metric Hit@3 on FB15k-237, MIT-KGC also ranked second and was slightly behind the SOTA model RESCAL-N3-RP [ 40 ], by 0.8%. This is because the PLM is mainly modelled from the semantic level and lacks the structural features of triples.…”

AutoWeird: Weird Translational Scoring Function Identified by Random Search

“…To analyze why AutoWeird is effective, we first count the occurrence of tail entities after adding inverse relations to the data set. Specifically, for each entity, we count how many times it appears as 188M 250 0.7989 ± 0.0004 0.7997 ± 0.0002 ComplEx [10] 188M 250 0.8095 ± 0.0007 0.8105 ± 0.0001 PairRE [2] 188M 200 0.8164 ± 0.0005 0.8172 ± 0.0005 AutoSF [17] 93.8M -0.8309 ± 0.0008 0.8317 ± 0.0007 TripleRE [15] 470M 200 0.8348 ± 0.0007 0.8360 ± 0.0006 ComplEx-RP [3] 188M 1000 0.8492 ± 0.0002 0.8497 ± 0.0002 AutoBLM+KGBench [18] 192M -0.8536 ± 0.0003 0.8548 ± 0.0002 AutoWeird 376M 500 0.6755 ± 0.0004 0.6752 ± 0.0003 EntOccur --0.3387 0.3387 the tail entity 2 in the training set. As is shown in Figure 1(a), a few number of entities has more than one million occurrences in the training set.…”