Accurate, Efficient and Scalable Graph Embedding

Abstract

The Graph Convolutional Network (GCN) model and its variants are powerful graph embedding tools for facilitating classification and clustering on graphs. However, a major challenge is to reduce the complexity of layered GCNs and make them parallelizable and scalable on very large graphs -state-of the art techniques are unable to achieve scalability without losing accuracy and efficiency. In this paper, we propose novel parallelization techniques for graph samplingbased GCNs that achieve superior scalable performance on very large graphs without compromising accuracy. Specifically, our GCN guarantees work-efficient training and produces order of magnitude savings in computation and communication. To scale GCN training on tightly-coupled shared memory systems, we develop parallelization strategies for the key steps in training: For the graph sampling step, we exploit parallelism within and across multiple sampling instances, and devise an efficient data structure for concurrent accesses that provides theoretical guarantee of near-linear speedup with number of processing units. For the feature propagation step within the sampled graph, we improve cache utilization and reduce DRAM communication by data partitioning. We prove that our partitioning strategy is a 2approximation for minimizing the communication time compared to the optimal strategy. We demonstrate that our parallel graph embedding outperforms state-of-the-art methods in scalability (with respect to number of processors, graph size and GCN model size), efficiency and accuracy on several large datasets. On a 40-core Xeon platform, our parallel training achieves 64× speedup (with AVX) in the sampling step and 25× speedup in the feature propagation step, compared to the serial implementation, resulting in a net speedup of 21×. Our scalable algorithm enables deeper GCN, as demonstrated by 1306× speedup on a 3-layer GCN compared to Tensorflow implementation of state-of-the-art.

GraphACT

“…One GCN layer based on G To accelerate GCN training, the work in [34] proposes parallelization techniques for multi-core platform. It partitions features to increase cache-hit of each core.…”

“…Ideally, a minibatch should be sampled such that all data required for gradient calculation (e.g., X (ℓ) of the minibatch nodes) fits in BRAM. Among the numerous algorithms [5,6,14,15,34], some return minibatches not suitable for hardware execution. We categorize these algorithms and analyze the hardware cost on external memory accesses.…”

“…The minibatch contains the same set of nodes V s for all the GCN layers. In other words, for each minibatch, [34] constructs a complete L-layer GCN from G s , with the layer nodes defined by V s and the layer connections defined by E s . Note that unlike [6,15], the graph sampler of [34] requires no information from node features.…”

“…Algorithms of low computation-communication ratio (i.e., [6,14,15]) impede the development of efficient hardware accelerators. For the remaining two algorithms, the complexity of the sampler of [34] is much lower than that of [5], and the training accuracy of [34] is higher. Thus, we select [34] as the target training algorithm for acceleration.…”

“…Remark on notation. Since we select the algorithm of [34], in the following sections, we mainly focus on GCN built on a subgraph G s . A symbol with an "s" subscript means it is related with a subgraph.…”