Efficient MPI‐AllReduce for large‐scale deep learning on GPU‐clusters

Abstract

Training models on large-scale GPUs-accelerated clusters are becoming a commonplace due to the increase in complexity and size in Deep Learning models. One of the main challenges for distributed training is the collective communication overhead for large message sizes: up to hundreds of MB. In this paper, we propose two hierarchical distributed memory multi-leader allreduce algorithms optimized for GPU-accelerated clusters (named lr_lr and lr_rab). In which, GPUs inside a computing node perform an intra-node communication phase to gather and store results of local reduced values to designated GPUs (known as node leaders). Node leaders then keep a role as an inter-node communicator. Each leader exchanges one part of reduced values to the leaders of the other nodes in parallel. Hence, we are capable of significantly reducing the time for injecting data into the inter-node network. We also overlap the inter-node and intra-node communication by implementing our proposal in a pipelined manner. We evaluate those algorithms on the discrete-event simulation Simgrid. We show that our algorithms, lr_lr and lr_rab, can cut down the execution time of an Allreduce micro-benchmark that uses the logical ring algorithm (lr) by up to 45% and 51%, respectively. With the pipelined implementation, our lr_lr_pipe achieve 15% performance improvement when compared to lr_lr. In addition, simulation result also project power savings for the network devices of up to 23% and 32%.

Allreduce algorithm optimization of OpenMPI communication library

“…MPI provides a powerful set of communication and synchronization mechanisms, enabling efficient communication and collaboration in parallel programs. Among these mechanisms, the Allreduce operation in MPI holds significant importance [2] . It is used for data reduction among multiple processes and is commonly employed for calculations such as summation and finding the maximum value.…”