NSCaching: Simple and Efficient Negative Sampling for Knowledge Graph Embedding

Abstract

Knowledge graph (KG) embedding is a fundamental problem in data mining research with many real-world applications. It aims to encode the entities and relations in the graph into low dimensional vector space, which can be used for subsequent algorithms. Negative sampling, which samples negative triplets from non-observed ones in the training data, is an important step in KG embedding. Recently, generative adversarial network (GAN), has been introduced in negative sampling. By sampling negative triplets with large scores, these methods avoid the problem of vanishing gradient and thus obtain better performance. However, using GAN makes the original model more complex and hard to train, where reinforcement learning must be used. In this paper, motivated by the observation that negative triplets with large scores are important but rare, we propose to directly keep track of them with cache. However, how to sample from and update the cache are two important questions. We carefully design the solutions, which are not only efficient but also achieve good balance between exploration and exploitation. In this way, our method acts as a "distilled" version of previous GANbased methods, which does not waste training time on additional parameters to fit the full distribution of negative triplets. The extensive experiments show that our method can gain significant improvement on various KG embedding models, and outperform the state-of-the-arts negative sampling methods based on GAN.

Attention-Based Adaptive Memory Network for Recommendation With Review and Rating

“…In training process, recent works mostly utilized random sampling, which treats each samples equally. However, in fact the importance of different samples are unequal [43], [45], [47]. Especially, the samples owning larger error between model's prediction values and ground truth, would play more important roles in performance evaluation.…”

Negative Sampling in Knowledge Representation Learning: A Mini-Review

“…Adversarial training is going on between the generator and the discriminator to optimize final knowledge representations. Reinforcement learning is required for training GAN [18]. The framework can be performed on various KRL models as it is independent of the specific form of the discriminator [16].…”

“…High-quality negative samples tend to get high plausibility measured by scoring functions. Motivated by the skewed score distribution of negative samples, Zhang et al [18]attempts to only track helpful and rare negatives of high plausibility with cache. NSCaching can be considered to be in the same group of GAN-based methods since they all parametrize the dynamic distribution of negative samples.…”

“…KBGAN [15] and IGAN [16] adversarially train the generator to provide better-quality negatives by applying a pretrained KRL model as the discriminator. TransE-SNS [17] and NSCaching [18] carry out negative sampling in a more concentrated way. Furthermore, enlightened by CKRL [19], NKRL [20] puts forward a confidence-aware negative sampling method.…”

Adaptive Network Alignment with Unsupervised and Multi-order Convolutional Networks

“…Networks, aka graphs, are structures that naturally capture relations between entities in data domains and information systems. Many applications, however, analyse not one, but multiple networks, e.g., for data discovery [12,26], social network analytics [33], knowledge graph reconciliation [50], and pattern matching in protein networks [29]. Network alignment, the task of pairing nodes between two isomorphic or nearisomorphic networks such that the paired nodes are similar w.r.t.…”