Sampling methods for efficient training of graph convolutional networks: A survey

Liu, Xin; Yan, Mingyu; Deng, Lei; Li, Guoqi; Ye, Xiaochun; Fan, Dongrui

doi:10.48550/arxiv.2103.05872

Cited by 3 publications

(4 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This way, each vertex has the same number of edges in the subgraphs, which reduces the computation complexity in GNN training and improves the regularity of the message aggregation for the subgraphs. It has been shown that the sampling-based methods can achieve accuracy competitive with the training of the full graph [14], [15], [16].…”

Section: Sampling-based Trainingmentioning

confidence: 99%

“…Sampling-based training. By using mini-batches, sampling-based training can scale to large graphs and has shown accuracy competitive with the full graph-based training [14], [15], [16]. GraphSage [11] first introduces the fixed-number vertex sampling and proposes the general message aggregation methods in GNN, such as sum, max pool, average, and LSTM.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

TurboGNN: Improving the End-to-End Performance for Sampling-Based GNN Training on GPUs

Shi

et al. 2023

IEEE Trans. Comput.

View full text Add to dashboard Cite

Graph Neural Networks (GNN) have evolved as powerful models for graph representation learning. Sampling-based training methods have been introduced to train large graphs without compromising accuracy. However, it is challenging for the existing GNN systems to effectively utilize multi-core accelerators, especially GPUs, due to a large number of atomic operations and unbalanced workload originating from the serial execution of multiple GNN processing stages. In this paper, we propose a combination of optimization techniques to accelerate the end-to-end performance of the sampling-based GNN training process. Specifically, we propose an adaptive share memory-based sampling technique and a degree-guided thread block scheduling strategy to optimize the graph sampling. Further, based on the observations of resource demand in different training stages, we propose an asynchronous pipeline-based scheduling method, which accelerates the GNN training by decoupling different training stages into a pipeline and therefore improves the GPU resource utilization significantly. The experimental results show that compared with the existing work, the proposed methods can achieve up to 5.6X performance speedup in the end-to-end performance.

show abstract

Section: Sampling-based Trainingmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

TurboGNN: Improving the End-to-End Performance for Sampling-Based GNN Training on GPUs

Shi

et al. 2023

IEEE Trans. Comput.

View full text Add to dashboard Cite

show abstract

“…After sampling, instead of using all neighbors as in the whole-graph training, sample-based training constructs a vertex's feature by only aggregating the features of the sampled set of neighbors. Existing sampling approaches largely fall into four categories: node-wise sampling, layer-wise sampling, subgraph-based sampling, and heterogeneous sampling [25]. They differ in the granularity of the sampling operation in one training minibatch.…”

Section: Sampling Algorithmsmentioning

confidence: 99%

“…Whole-graph training introduces inherent coordination and communication overheads that are hard to overcome as the system scales. Scaling sample-based training, on the other hand, requires (a) sampling algorithms that can form mini-batches without incurring into the "neighbor explosion" problem [14,25] and (b) scalable systems to execute these sampling algorithms efficiently. We review recent research that addresses these requirements.…”

Section: Introductionmentioning

confidence: 99%

Scalable Graph Neural Network Training: The Case for Sampling

Serafini,

Guan

2021

Preprint

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) are a new and increasingly popular family of deep neural network architectures to perform learning on graphs. Training them efficiently is challenging due to the irregular nature of graph data. The problem becomes even more challenging when scaling to large graphs that exceed the capacity of single devices. Standard approaches to distributed DNN training, such as data and model parallelism, do not directly apply to GNNs. Instead, two different approaches have emerged in the literature: whole-graph and sample-based training.In this paper, we review and compare the two approaches. Scalability is challenging with both approaches, but we make a case that research should focus on sample-based training since it is a more promising approach. Finally, we review recent systems supporting sample-based training.

show abstract