Reducing Communication in Graph Neural Network Training

Tripathy, Aruna; Yelick, Katherine; Buluç, Aydın

doi:10.1109/sc41405.2020.00074

Cited by 47 publications

(62 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though advances in sparse/irregular tensor processing [34] and graph processing [63,154] may prove useful in accelerating GNNs, addressing their unique computing challenges requires more specialized proposals. Some attempts have been done from a software perspective, i.e., adapting the GNN operations to better match the capabilities of CPUs or GPUs [106,144,155]; and from a hardware perspective, i.e., designing custom processors tailored to the demands of GNNs [7,53,103,164]. However, recent surveys and reviews [11,16,19,66,91,160,181,185] lack a comprehensive analysis of such advances.…”

Section: Deep Learning On Graphsmentioning

confidence: 99%

Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

et al. 2021

View full text Add to dashboard Cite

Graph Neural Networks (GNNs) have exploded onto the machine learning scene in recent years owing to their capability to model and learn from graph-structured data. Such an ability has strong implications in a wide variety of fields whose data are inherently relational, for which conventional neural networks do not perform well. Indeed, as recent reviews can attest, research in the area of GNNs has grown rapidly and has lead to the development of a variety of GNN algorithm variants as well as to the exploration of ground-breaking applications in chemistry, neurology, electronics, or communication networks, among others. At the current stage research, however, the efficient processing of GNNs is still an open challenge for several reasons. Besides of their novelty, GNNs are hard to compute due to their dependence on the input graph, their combination of dense and very sparse operations, or the need to scale to huge graphs in some applications. In this context, this article aims to make two main contributions. On the one hand, a review of the field of GNNs is presented from the perspective of computing. This includes a brief tutorial on the GNN fundamentals, an overview of the evolution of the field in the last decade, and a summary of operations carried out in the multiple phases of different GNN algorithm variants. On the other hand, an in-depth analysis of current software and hardware acceleration schemes is provided, from which a hardware-software, graph-aware, and communication-centric vision for GNN accelerators is distilled.

show abstract

Section: Deep Learning On Graphsmentioning

confidence: 99%

Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Various distributed-memory parallel sparse matrix times tallskinny dense matrix (SpMM) algorithms are buried in application papers. In particular, MPI_FAUN [20] implements the 1.5D algorithm we use in our paper and CAGNET [30] implements the bulksynchronous 2D algorithm presented here. Neither work study the impacts of partitioning strategy in local computation costs as we do here.…”

Section: Related Workmentioning

confidence: 99%

“…We consider a wide variety of sparse matrix inputs that represent problems from graph neural networks (reddit and amazon [30]), eigensolvers in nuclear structure calculations (nm7 and nm8 [4]), low-rank or non-negative matrix factorization (com-Orkut [20]), and bioinformatics (isolates [6]). Various properties of these matrices are presented in Table 2.…”

Section: Evaluation 61 Setupmentioning

confidence: 99%

“…However, the primary revolution that made research into SpMM explode in the last few years is the success of Graph Neural Networks (GNN) in a variety of supervised and semi-supervised learning scenarios. As shown in recent publications, SpMM is the workhorse of training GNNs [18,19,30]. In addition, several groups are also uncovering the potential of exploiting sparsity in convolutional and recurrent deep neural networks (DNNs) [14].…”

Section: Introductionmentioning

confidence: 99%

“…We discuss these differences in detail in Section 3. Algorithmic papers targeting higher level problems such as non-negative matrix factorization [20] and graph neural network training [30] implemented and studied their own subset of parallel SpMM algorithms.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed-memory parallel algorithms for sparse times tall-skinny-dense matrix multiplication

Selvitopi

Brock

Nisa

et al. 2021

Proceedings of the ACM International Conference on Supercomputing

Self Cite

View full text Add to dashboard Cite

Sparse times dense matrix multiplication (SpMM) finds its applications in well-established fields such as computational linear algebra as well as emerging fields such as graph neural networks. In this study, we evaluate the performance of various techniques for performing SpMM as a distributed computation across many nodes by focusing on GPU accelerators. We examine how the actual local computational performance of state-of-the-art SpMM implementations affect computational efficiency as dimensions change when we scale to large numbers of nodes, which proves to be an unexpectedly important bottleneck. We consider various distribution strategies, including A-Stationary, B-Stationary, and C-Stationary algorithms, 1.5D and 2D algorithms, and RDMA-based and bulk synchronous methods of data transfer. Our results show that the best choice of algorithm and implementation technique depends not only on the cost of communication for particular matrix sizes and dimensions, but also on the performance of local SpMM operations. Our evaluations reveal that with the involvement of GPU accelerators, the best design choices for SpMM differ from the conventional algorithms that are known to perform well for dense matrix-matrix or sparse matrix-sparse matrix multiplies. CCS CONCEPTS• Computing methodologies → Parallel algorithms.

show abstract

OpenGraphGym: A Parallel Reinforcement Learning Framework for Graph Optimization Problems

Zheng

Wang

Song

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper presents an open-source, parallel AI environment (named OpenGraphGym) to facilitate the application of reinforcement learning (RL) algorithms to address combinatorial graph optimization problems. This environment incorporates a basic deep reinforcement learning method, and several graph embeddings to capture graph features, it also allows users to rapidly plug in and test new RL algorithms and graph embeddings for graph optimization problems. This new opensource RL framework is targeted at achieving both high performance and high quality of the computed graph solutions. This RL framework forms the foundation of several ongoing research directions, including 1) benchmark works on different RL algorithms and embedding methods for classic graph problems; 2) advanced parallel strategies for extreme-scale graph computations, as well as 3) performance evaluation on real-world graph solutions.

show abstract

Reducing Communication in Graph Neural Network Training

Cited by 47 publications

References 39 publications

Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

Computing Graph Neural Networks: A Survey from Algorithms to Accelerators

Distributed-memory parallel algorithms for sparse times tall-skinny-dense matrix multiplication

OpenGraphGym: A Parallel Reinforcement Learning Framework for Graph Optimization Problems

Contact Info

Product

Resources

About