Optimizing Sparse Matrix Multiplications for Graph Neural Networks

Qiu, Shenghao; Liang, You; Wang, Zheng

doi:10.1007/978-3-030-99372-6_7

Cited by 10 publications

(6 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…New DNN workloads exhibit new characteristics. For example, inputs to the graph neural network are sparse matrices [40] scheduling [39], [41], [42]. These approaches are complementary to AIACC-Training.…”

Section: Discussionmentioning

confidence: 99%

AIACC-Training: Optimizing Distributed Deep Learning Training through Multi-streamed and Concurrent Gradient Communications

Qiu

et al. 2022

2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

There is a growing interest in training deep neural networks (DNNs) in a GPU cloud environment. This is typically achieved by running parallel training workers on multiple GPUs across computing nodes. Under such a setup, the communication overhead is often responsible for long training time and poor scalability. This paper presents AIACC-Training, a unified communication framework designed for the distributed training of DNNs in a GPU cloud environment. AIACC-Training permits a training worker to participate in multiple gradient communication operations simultaneously to improve network bandwidth utilization and reduce communication latency. It employs auto-tuning techniques to dynamically determine the right communication parameters based on the input DNN workloads and the underlying network infrastructure. AIACC-Training has been deployed to production at Alibaba GPU Cloud with 3000+ GPUs executing AIACC-Training optimized code at any time. Experiments performed on representative DNN workloads show that AIACC-Training outperforms existing solutions, improving the training throughput and scalability by a large margin.

show abstract

Section: Discussionmentioning

confidence: 99%

AIACC-Training: Optimizing Distributed Deep Learning Training through Multi-streamed and Concurrent Gradient Communications

Qiu

et al. 2022

2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

show abstract

“… Matrix view : Expressing GNN models from a coarse‐grained, global perspective, emphasizing operations involving sparse adjacency matrices and feature vectors. Both views are essential tools for studying GNNs and complement each other 15,78,80 (Figure 9B). …”

Section: Large‐scale Glmsmentioning

confidence: 99%

Accelerating the integration of ChatGPT and other large‐scale AI models into biomedical research and healthcare

Wang

Feng

et al. 2023

MedComm – Future Medicine

View full text Add to dashboard Cite

Large-scale artificial intelligence (AI) models such as ChatGPT have the potential to improve performance on many benchmarks and real-world tasks. However, it is difficult to develop and maintain these models because of their complexity and resource requirements. As a result, they are still inaccessible to healthcare industries and clinicians. This situation might soon be changed because of advancements in graphics processing unit (GPU) programming and parallel computing. More importantly, leveraging existing large-scale AIs such as GPT-4 and Med-PaLM and integrating them into multiagent models (e.g., Visual-ChatGPT) will facilitate real-world implementations. This review aims to raise awareness of the potential applications of these models in healthcare. We provide a general overview of several advanced large-scale AI models, including language models, vision-language models, graph learning models, language-conditioned multiagent models, and multimodal embodied models. We discuss their potential medical applications in addition to the challenges and future directions. Importantly, we stress the need to align these models with human values and goals, such as using reinforcement learning from human feedback, to ensure that they provide accurate and personalized insights that support human decisionmaking and improve healthcare outcomes.

show abstract

“…Sparse matrix-vector multiplication (SpMV) multiplies the adjacency matrix 𝐴 (transposed) of graph 𝐺 with a vector 𝑥 of values, one per vertex; it is used in Graph/Recurrent Neural Networks [40], Topic Search [25], and Belief Propagation [22]. In this paper, we consider MIP algorithms over a single thread, and study how to compute the answers to multiple sources in one go, in an interleaved manner.…”

Section: Preliminariesmentioning

confidence: 99%

MITra: A Framework for Multi-Instance Graph Traversal

Li¹,

Zhao

Ntarmos³

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

This paper presents MITra, a framework for composing multi-instance graph algorithms that traverse from multiple source vertices simultaneously over a single thread. Underlying MITra is a model of multi-instance traversal that uniformly captures traversal sharing across instances. Based on this, MITra provides a programming model that allows users to express traversals by declaring vertex ranks and specify computation logic via an edge function. It synthesizes multi-instance traversal algorithms from declared vertex ranks and edge functions adopted from classic single-instance algorithms, automatically sharing computation across instances and benefiting from SIMD. We show that MITra can generate multi-instance algorithms provably better than existing ones, while being more expressive than traditional frameworks. In addition to the ease of programming, we experimentally verify that MITra is on average an order of magnitude faster than approaches based on existing frameworks for common graph algorithms, and is comparable to the state-of-the-art highly optimized one-off algorithms.

show abstract

Optimizing Sparse Matrix Multiplications for Graph Neural Networks

Abstract: This is a repository copy of Optimizing Sparse Matrix Multiplications for Graph Neural Networks.

Cited by 10 publications

References 41 publications

AIACC-Training: Optimizing Distributed Deep Learning Training through Multi-streamed and Concurrent Gradient Communications

AIACC-Training: Optimizing Distributed Deep Learning Training through Multi-streamed and Concurrent Gradient Communications

Accelerating the integration of ChatGPT and other large‐scale AI models into biomedical research and healthcare

MITra: A Framework for Multi-Instance Graph Traversal

Contact Info

Product

Resources

About