Synthesizing Collective Communication Algorithms for Heterogeneous Networks with TACCL

Shah, Aashaka; Chidambaram, Vijay; Cowan, Meghan; Maleki, Saeed; Musuvathi, Madan; Mytkowicz, Todd; Nelson, Jacob L.; Saarikivi, Olli; Singh, Rachee

doi:10.48550/arxiv.2111.04867

Cited by 2 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Collective Communication on Heterogeneous Clusters. TACCL [39] models communication as an mixed integer linear programming problem and finds routing and scheduling of each data chunk to minimize communication. BlueConnect [8] decomposes All-Reduce to fit into heterogeneous network hierarchy.…”

Section: Related Workmentioning

confidence: 99%

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

Zhang,

Diao,

et al. 2024

Proceedings of the Nineteenth European Conference on Computer Systems

View full text Add to dashboard Cite

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents HAP, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. HAP jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A*-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, HAP explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that HAP achieves up to 2.41x speed-up on heterogeneous clusters. CCS Concepts:• Computing methodologies → Distributed computing methodologies.

show abstract

Section: Related Workmentioning

confidence: 99%

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

Zhang,

Diao,

et al. 2024

Proceedings of the Nineteenth European Conference on Computer Systems

View full text Add to dashboard Cite

show abstract

“…Chunks. Communication data is usually broken into multiple chunks [32,47,56] and then these chunks are fed into this 2×Dstage pipeline to keep all dimensions busy. A chunk is a portion of data to participate in the collective, and the collective algorithm can work on each chunk independently.…”

Section: Multi-rail Hierarchical Collective Comm Algorithmsmentioning

confidence: 99%

“…On the other hand, step_latency is determined by the network component latencies (e.g., NIC latency, link latency, etc.) when transferring a minimumsize message between two NPUs [56]. On real systems, 𝐴 𝐾 can be calculated by running a minimum size collective on dimK.…”

Section: Understanding All Latency Parametersmentioning

confidence: 99%

“…The Aggr BW/NPU is the product of the BW/link determined by the technology and Links/NPU is determined by the topology. **Link Technologies for each Dimension: chiplet-to-chiplet (within a package) [14,25,54,57], package-to-package (within a server node) [7,22,56,65], node-to-node [7,16,35,56], pod-to-pod. [8,16,35,56].…”

Section: Inter-dimension Schedule Consistencymentioning

confidence: 99%

“…**Link Technologies for each Dimension: chiplet-to-chiplet (within a package) [14,25,54,57], package-to-package (within a server node) [7,22,56,65], node-to-node [7,16,35,56], pod-to-pod. [8,16,35,56]. Note -for all topologies, the last dimension uses NICs, which is node-to-node for 2D, and pod-to-pod for 3D and 4D.…”

Section: Inter-dimension Schedule Consistencymentioning

confidence: 99%

See 2 more Smart Citations

Themis

Rashidi

Won

Srinivasan

et al. 2022

Proceedings of the 49th Annual International Symposium on Computer Architecture

View full text Add to dashboard Cite

Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (e.g., GPU/TPU). However, distributed training adds communication overhead between the NPUs in order to synchronize the gradients and/or activation, depending on the parallelization strategy. In next-generation platforms for training at scale, NPUs will be connected through multidimensional networks with diverse, heterogeneous bandwidths. This work identifies a looming challenge of keeping all network dimensions busy and maximizing the network BW within the hybrid environment if we leverage scheduling techniques for collective communication on systems today. We propose Themis, a novel collective scheduling scheme that dynamically schedules collectives (divided into chunks) to balance the communication loads across all dimensions, further improving the network BW utilization. Our results show that on average, Themis can improve the network BW utilization of the single All-Reduce by 1.72× (2.70× max), and improve the end-to-end training iteration performance of real workloads such as ResNet-152, GNMT, DLRM, and Transformer-1T by 1.49× (2.25× max), 1.30× (1.78× max), 1.30× (1.77× max), and 1.25× (1.53× max), respectively.

show abstract

Synthesizing Collective Communication Algorithms for Heterogeneous Networks with TACCL

Cited by 2 publications

References 10 publications

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

Themis

Contact Info

Product

Resources

About