Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Heterogeneous Graphs

Zheng, Da; Song, Xiang; Yang, Chengru; LaSalle, Dominique; Karypis, George

doi:10.1145/3534678.3539177

Cited by 23 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zheng et al [142] formulate the graph partition problem in the DistDGL as a multi-constraint partition problem that aims at balancing training/validation/test vertices/edges in each partition. They adopt the multi-constraint mechanism in METIS to realize the customized objective of graph partition for both homogeneous graphs and heterogeneous graphs [143], and further expand this mechanism to a two-level partitioning strategy that handles the situation with multi-GPUs and multi-machines and balances the computation. Lin et al [67] apply streaming graph partition -Linear Deterministic Greedy (LGD) [89] and use a new affinity score (E.q.…”

Section: Graph Partition In Gnnmentioning

confidence: 99%

“…Operator-parallel execution model enables the inter-batch parallelism and generates multi-computation graph in parallel by mixing the execution of all operators in the pipeline. Zheng et al [143] decompose the computation graph generation into multiple stages (e.g., graph sampling, neighborhood subgraph construction, feature extraction) and each stage can be treated as an operator. Dependencies exist among different operators, while there are no dependencies among mini-batches.…”

Section: Execution Model Of Computation Graph Generationmentioning

confidence: 99%

“…DistDGLv2 [143] is a continuous work of DistDGL by the DGL team from Amazon. It takes the characteristic of the heterogeneous graph into consideration for balanced graph partition.…”

Section: Systems On Gpu Clustersmentioning

confidence: 99%

“…Paddle Graph Learning 2 framework provides easy and fast programming of the message passing-based graph learning on heterogeneous graphs with distributed computing. DistDGLv2 [143] takes the imbalanced workload partition into consideration and leverages the multi-constraint technique in METIS to mitigate this problem. More attention should be paid to this research topic to address the above problems.…”

Section: Future Directionmentioning

confidence: 99%

See 3 more Smart Citations

Distributed Graph Neural Network Training: A Survey

Shao¹,

Li²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) are a type of deep learning models that learning over graphs, and have been successfully applied in many domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs, since it is able to provide abundant computing resources. However, the dependency of graph structure increases the difficulty of achieving high-efficiency distributed GNN training, which suffers from the massive communication and workload imbalance. In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed. Yet, there is a lack of systematic review on the optimization techniques from graph processing to distributed execution. In this survey, we analyze three major challenges in distributed GNN training that are massive feature communication, the loss of model accuracy and workload imbalance. Then we introduce a new taxonomy for the optimization techniques in distributed GNN training that address the above challenges. The new taxonomy classifies existing techniques into four categories that are GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. We carefully discuss the techniques in each category. In the end, we summarize existing distributed GNN systems for multi-GPUs, GPU-clusters and CPU-clusters, respectively, and give a discussion about the future direction on scalable GNNs.

show abstract

Section: Graph Partition In Gnnmentioning

confidence: 99%

Section: Execution Model Of Computation Graph Generationmentioning

confidence: 99%

“…DistDGLv2 [143] is a continuous work of DistDGL by the DGL team from Amazon. It takes the characteristic of the heterogeneous graph into consideration for balanced graph partition.…”

Section: Systems On Gpu Clustersmentioning

confidence: 99%

Section: Future Directionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed Graph Neural Network Training: A Survey

Shao¹,

Li²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To reduce computational and memory costs of GNN training, many GNN systems consider a sample of neighbors in a minibatch (a batch of training vertices instead of all vertices) to reduce the memory and computational load. Many systems adopt this execution model because it has been observed to have a positive effect on training [19,36] and because the smaller memory footprint allows the use of GPUs which have limited memory compared to CPUs.…”

Section: Introductionmentioning

confidence: 99%

Efficient Neighbor-Sampling-based GNN Training on CPU-FPGA Heterogeneous Platform

Zhang

Kuppannagari

Kannan³

et al. 2021

2021 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

We present BatchGNN, a distributed CPU system that showcases techniques that can be used to efficiently train GNNs on terabyte-sized graphs. It reduces communication overhead with macrobatching in which multiple minibatches' subgraph sampling and feature fetching are batched into one communication relay to reduce redundant feature fetches when input features are static. BatchGNN provides integrated graph partitioning and native GNN layer implementations to improve runtime, and it can cache aggregated input features to further reduce sampling overhead. BatchGNN achieves an average 3× speedup over DistDGL on three GNN models trained on OGBN graphs, outperforms the runtimes reported by distributed GPU systems P 3 and DistDGLv2, and scales to a terabyte-sized graph. * Bo Wu contributed to this work while employed at KatanaGraph. Preprint. Under review.

show abstract