Practical Near-Data-Processing Architecture for Large-Scale Distributed Graph Neural Network

Abstract

Graph Neural Networks have drawn tremendous attention in the past few years due to their convincing performance and high interpretability in various graph-based tasks like link prediction and node classification. With the ever-growing graph size in the real world, especially for industrial graphs at a billionlevel, the storage of graphs can easily consume Terabytes so that the process of GNNs has to be processed in a distributed manner. As a result, the execution could be inefficient due to the expensive cross-node communication and irregular memory access. Various GNN accelerators have been proposed for efficient GNN processing. They, however, mainly focused on small and medium-size graphs, which is not applicable to large-scale distributed graphs. In this paper, we present a practical Near-Data-Processing architecture based on a memory-pool system for large-scale distributed GNNs. We propose a customized memory fabric interface to construct the memory pool for low-latency and high throughput cross-node communication, which can provide flexible memory allocation and strong scalability. A practical Near-Data-Processing design is proposed for efficient work offloading and bandwidth utilization improvement. Moreover, we also introduce a partition and scheduling scheme to further improve performance and achieve workload balance. Comprehensive evaluations demonstrate that the proposed architecture can achieve up to 27× and 8× higher training speed compared to two state-of-the-art distributed GNN frameworks: Deep Graph Library and P 3 , respectively.INDEX TERMS Graph neural network, large-scale graph processing, memory pool, near data processing.

Hyperscale FPGA-as-a-service architecture for large-scale distributed graph neural network

“…Others such as GCNAX [44], BoostGCN [77], Rubik [14], GraphACT [76], GNNSampler [47], and Grip [39,40] optimizing data reuse are not applicable to LSD-GNN either, since the chance to find reuse within 512-node mini-batch compared with 10+ billion total nodes is extremely low. Huang et al [34] works on a similar problem like this paper, but under a different disaggregated memory pool context. Sparse tensor hardware: Architecture innovations from sparse tensor accelerators [32,58,62,81] can be adopted to solve the SM-GNN problems, since the full-batch training of SM-GNN can be well processed with sparseMM on a single machine.…”