Efficient sparse collective communication and its application to accelerate distributed deep learning

Abstract

Efficient collective communication is crucial to parallel-computing applications such as distributed training of large-scale recommendation systems and natural language processing models. Existing collective communication libraries focus on optimizing operations for dense inputs, resulting in transmissions of many zeros when inputs are sparse. This counters current trends that see increasing data sparsity in large models.We propose OmniReduce, an efficient streaming aggregation system that exploits sparsity to maximize effective bandwidth use by sending only non-zero data blocks. We demonstrate that this idea is beneficial and accelerates distributed training by up to 8.2×. Even at 100 Gbps, OmniReduce delivers 1.4-2.9× better performance for network-bottlenecked DNNs. CCS CONCEPTS• Computer systems organization → Distributed architectures; • Computing methodologies → Machine learning.

Flare: Flexible In-Network Allreduce

“…Last, some solutions target programmable switches [11,24,25], that provide more flexibility by allowing the programmer to specify the type of processing to be executed on the packets with high-level programming languages. Programmable switches are often implemented through Reconfigurable Match-Action Tables (RMTs) [26][27][28][29], and can be configured with the P4 programming language [30][31][32].…”

“…F2 -Sparse data. Many applications need to reduce sparse data [5,25,35,39], i.e., data containing mostly null values. To save bandwidth and improve performance, an application might only transmit and reduce the non-null values.…”

“…However, to the best of our knowledge, none of the solutions targeting fixed-function switches provide explicit support for sparse data ( ). Among the programmable switches solutions, only one partially targets innetwork sparse data reduction [25]. However, it forces the application to sparsify the data per-block, i.e., to send sparse blocks of dense data ( ).…”

“…However, it forces the application to sparsify the data per-block, i.e., to send sparse blocks of dense data ( ). This is however not possible for any application and, even when possible (for example for deep learning models training), this negatively affects the convergence of the training [25]. On the other hand, with Flare we design the first in-network sparse allreduce algorithm, that does not make any assumption on the data sparsity, and can process the data as generated by the hosts ( ), improving application performance compared to existing solutions (Section 7.1).…”

“…Each switch can participate simultaneously in different allreduces (issued by the same user/application or by different ones), and the network manager assigns a unique identifier to each allreduce, so that only packets belonging to the same allreduce are aggregated together. Because each allreduce consumes some memory resources on the switch, we assume that the memory is statically partitioned across a predefined maximum number of allreduces, as done by most in-network reduction approaches [9,11,[16][17][18]25]. If any switch on the reduction tree is already managing the maximum possible number of allreduces, the network manager can try to recompute a different reduction tree excluding that switch or just reject the request issued by the application, which then needs to rely on host-based allreduce [9,16].…”