Fast and Accurate Network Embeddings via Very Sparse Random Projection

Abstract

We present FastRP, a scalable and performant algorithm for learning distributed node representations in a graph. FastRP is over 4,000 times faster than state-of-the-art methods such as DeepWalk and node2vec, while achieving comparable or even better performance as evaluated on several real-world networks on various downstream tasks. We observe that most network embedding methods consist of two components: construct a node similarity matrix and then apply dimension reduction techniques to this matrix. We show that the success of these methods should be attributed to the proper construction of this similarity matrix, rather than the dimension reduction method employed.FastRP is proposed as a scalable algorithm for network embeddings. Two key features of FastRP are: 1) it explicitly constructs a node similarity matrix that captures transitive relationships in a graph and normalizes matrix entries based on node degrees; 2) it utilizes very sparse random projection, which is a scalable optimization-free method for dimension reduction. An extra benefit from combining these two design choices is that it allows the iterative computation of node embeddings so that the similarity matrix need not be explicitly constructed, which further speeds up FastRP. FastRP is also advantageous for its ease of implementation, parallelization and hyperparameter tuning. The source code is available at https://github.com/GTmac/FastRP.

QUINT: Node embedding using network hashing

“…Recently, sketching-based techniques have been explored for generating node embeddings of large networks. Chen et al [36] proposed an algorithm for computing node embedding of large networks using very sparse random projections [37]. FREDE [38] generates linear space embedding of nodes using deterministic matrix sketching [39], and InstantEmbedding [40] computes node embedding using local PageRank computation.…”

Subset Node Representation Learning over Large Dynamic Graphs

“…It then obtains an updated embedding vector of 𝑠 by the following three steps: 1) It first updates the estimate vector 𝒑 𝑡 𝑠 and 𝒓 𝑡 𝑠 from Line 2 to Line 9; 2) It then calls the forward local push method to obtain the updated estimations, 𝒑 𝑡 𝑠 ; 3) We then use the hash kernel projection step to get an updated embedding. This projection step is from InstantEmbedding where two universal hash functions are defined as ℎ 𝑑 : N → [𝑑] and ℎ sgn : N → {±1} 6 . Then the hash kernel based on these two hash functions is defined as 𝐻 ℎ sgn ,ℎ 𝑑 (𝒙) : R 𝑛 → R 𝑑 where each entity 𝑖 is 𝑗 ∈ℎ −1 𝑑 (𝑖) 𝑥 𝑗 ℎ sgn ( 𝑗).…”

“…Then the hash kernel based on these two hash functions is defined as 𝐻 ℎ sgn ,ℎ 𝑑 (𝒙) : R 𝑛 → R 𝑑 where each entity 𝑖 is 𝑗 ∈ℎ −1 𝑑 (𝑖) 𝑥 𝑗 ℎ sgn ( 𝑗). Different from random projection used in RandNE [47] and FastRP [6], hash functions has O (1) memory cost while random projection based method has O (𝑑𝑛) if the Gaussian matrix is used. Furthermore, hash kernel keeps unbiased estimator for the inner product [42].…”

Fast Attributed Multiplex Heterogeneous Network Embedding

“…There are several variants of random projection techniques for manifold and network learning [14,20]. Recently, RandNE [34] and FastRP [3] are proposed to capture high-order structure information of homogeneous network by Gaussian random projection and sparse random projection, respectively. However, these methods ignore the heterogeneity and attributes of nodes and relations, and thus cannot capture the rich semantics on AMHENs.…”

“…The product attributes include the price, sales-rank, brand, category, etc. AMiner dataset 3 contains three types of nodes: author, paper and conference. The domain of papers is considered as the class label.…”