A New Model-Based Approach to Performance Comparison of MPI Collective Algorithms

Nuriyev, Emin; Lastovetsky, Alexey

doi:10.1007/978-3-030-86359-3_2

Cited by 2 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rashti M. J. et al [5] utilized graph embedding and module discovery of application communication topologies to match them with the physical topologies of multi-core clusters with multi-level networks. Nuriyev E. et al [6] employed an analytical modeling approach and derived more detailed models from the code implementation of the algorithms rather than their high-level mathematical definitions. They estimated the model parameters for each collective algorithm and incorporated the execution of the algorithm into the corresponding communication experiments.…”

Section: Introductionmentioning

confidence: 99%

Allreduce algorithm optimization of OpenMPI communication library

Zhang,

Wan,

2024

Third International Conference on Algorithms, Microchips, and Network Applications (AMNA 2024)

View full text Add to dashboard Cite

Section: Introductionmentioning

confidence: 99%

Allreduce algorithm optimization of OpenMPI communication library

Zhang,

Wan,

2024

Third International Conference on Algorithms, Microchips, and Network Applications (AMNA 2024)

View full text Add to dashboard Cite

Analyzing the impact of the MPI allreduce in distributed training of convolutional neural networks

et al. 2022

View full text Add to dashboard Cite

For many distributed applications, data communication poses an important bottleneck from the points of view of performance and energy consumption. As more cores are integrated per node, in general the global performance of the system increases yet eventually becomes limited by the interconnection network. This is the case for distributed data-parallel training of convolutional neural networks (CNNs), which usually proceeds on a cluster with a small to moderate number of nodes. In this paper, we analyze the performance of the Allreduce collective communication primitive, a key to the efficient data-parallel distributed training of CNNs. Our study targets the distinct realizations of this primitive in three high performance instances of Message Passing Interface (MPI), namely MPICH, OpenMPI, and IntelMPI, and employs a cluster equipped with state-of-the-art processor and network technologies. In addition, we apply the insights gained from the experimental analysis to the optimization of the TensorFlow framework when running on top of Horovod. Our study reveals that a careful selection of the most convenient MPI library and Allreduce (ARD) realization accelerates the training throughput by a factor of 1.2× compared with the default algorithm in the same MPI library, and up to 2.8× when comparing distinct MPI libraries in a number of relevant combinations of CNN model+dataset.

show abstract

A New Model-Based Approach to Performance Comparison of MPI Collective Algorithms

Cited by 2 publications

References 17 publications

Allreduce algorithm optimization of OpenMPI communication library

Allreduce algorithm optimization of OpenMPI communication library

Analyzing the impact of the MPI allreduce in distributed training of convolutional neural networks

Contact Info

Product

Resources

About