Optimization of Collective Communication Operations in MPICH

Abstract

We describe our work on improving the performance of collective communication operations in MPICH for clusters connected by switched networks. For each collective operation, we use multiple algorithms depending on the message size, with the goal of minimizing latency for short messages and minimizing bandwidth use for long messages. Although we have implemented new algorithms for all MPI (Message Passing Interface) collective operations, because of limited space we describe only the algorithms for allgather, broadcast, all-to-all, reduce-scatter, reduce, and allreduce. Performance results on a Myrinet-connected Linux cluster and an IBM SP indicate that, in all cases, the new algorithms significantly outperform the old algorithms used in MPICH on the Myrinet cluster, and, in many cases, they outperform the algorithms used in IBM's MPI on the SP. We also explore in further detail the optimization of two of the most commonly used collective operations, allreduce and reduce, particularly for long messages and nonpower-of-two numbers of processes. The optimized algorithms for these operations perform several times better than the native algorithms on a Myrinet cluster, IBM SP, and Cray T3E. Our results indicate that to achieve the best performance for a collective communication operation, one needs to use a number of different algorithms and select the right algorithm for a particular message size and number of processes.

Bandwidth optimal all-reduce algorithms for clusters of workstations

“…In this paper, we use the existing results in various papers (e.g. [7,22,25]) as the components of our proposed algorithm. Our contribution lies in showing that putting these existing components in the particular way yields a contention-free bandwidth optimal all-reduce algorithm for the tree topology.…”

“…Many efficient platform independent algorithms for this operation have been proposed [22,23,25,27]. In [22], Rabenseifner…”

“…The limitation of this algorithm is that it is difficult to realize the butterfly communication pattern without incurring network contention in contemporary SMP/multi-core clusters. Many all-gather [7,24,25] and reduce-scatter [13,24,25] algorithms, which are parts of the proposed all-reduce algorithm, have been developed. The technique for finding contention-free logical ring on tree topologies was developed in [7].…”

“…Our proposed bandwidth optimal all-reduce algorithm combines three existing ideas: (1) realizing the all-reduce operation by a reduce-scatter operation followed by an all-gather operation [22], (2) realizing both the reduce-scatter operation and the all-gather operation using logical ring based algorithms [7,25], and (3) constructing a contention-free logical ring on the tree topology [7]. Our contribution lies in combining these three components in an all-reduce algorithm and in showing that the resulting all-reduce algorithm is bandwidth optimal.…”

“…The optimality is achieved by using logical ring based reduce-scatter and all-gather algorithms, and by finding a contention-free logical ring on the tree topology. The logical ring all-gather algorithm is described in [25] and the technique to compute a contention-free logical ring on a tree topology can be found in [7]. Next, we will describe the logical ring reduce-scatter scheme.…”