Efficient Barrier and Allreduce on Infiniband clusters using multicast and adaptive algorithms

Mamidala, Amith R.; Liu, Jiuxing; Panda, Dhabaleswar K.

doi:10.1109/clustr.2004.1392611

Cited by 28 publications

(18 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The butterfly-like algorithm has been developed some times ago [22,27] and has been extended to handle non-power-of-two numbers of processes [23]. Various architecture specific all-reduce schemes have also been developed [1,4,12,17,26]. An all-reduce algorithm was designed for BlueGene/L systems in [1].…”

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

“…In [12], an all-reduce scheme that takes advantage of remote DMA (RDMA) capability was developed for VIA-based clusters. The work in [17] investigated an adaptive all-reduce algorithm in an InfiniBand cluster that deals with the situation when not all nodes arrive at the call site at the same time. A study on the all-reduce operation over WAN can be found in [4].…”

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Bandwidth optimal all-reduce algorithms for clusters of workstations

Patarasuk

Yuan

2009

Journal of Parallel and Distributed Computing

312

147

View full text Add to dashboard Cite

We consider an efficient realization of the all-reduce operation with large data sizes in cluster environments, under the assumption that the reduce operator is associative and commutative. We derive a tight lower bound of the amount of data that must be communicated in order to complete this operation and propose a ring-based algorithm that only requires tree connectivity to achieve bandwidth optimality. Unlike the widely used butterfly-like all-reduce algorithm that incurs network contention in SMP/multi-core clusters, the proposed algorithm can achieve contention-free communication in almost all contemporary clusters including SMP/multi-core clusters and Ethernet switched clusters with multiple switches. We demonstrate that the proposed algorithm is more efficient than other algorithms on clusters with different nodal architectures and networking technologies when the data size is sufficiently large.

show abstract

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

Bandwidth optimal all-reduce algorithms for clusters of workstations

Patarasuk

Yuan

2009

Journal of Parallel and Distributed Computing

312

147

View full text Add to dashboard Cite

show abstract

“…Example algorithms include the all-reduce and barrier algorithms in [17] and the broadcast and reduce algorithms in [28]. Our work advocates further development of such algorithms as well as other mechanisms to handle the imbalanced process arrival pattern problem.…”

Section: Related Workmentioning

confidence: 99%

A study of process arrival patterns for MPI collective operations

Faraj

Patarasuk

Yuan

2007

Proceedings of the 21st Annual International Conference on Supercomputing

View full text Add to dashboard Cite

Process arrival pattern, which denotes the timing when different processes arrive at an MPI collective operation, can have a significant impact on the performance of the operation. In this work, we characterize the process arrival patterns in a set of MPI programs on two common cluster platforms, use a micro-benchmark to study the process arrival patterns in MPI programs with balanced loads, and investigate the impacts of different process arrival patterns on collective algorithms. Our results show that (1) the differences between the times when different processes arrive at a collective operation are usually sufficiently large to affect the performance; (2) application developers in general cannot effectively control the process arrival patterns in their MPI programs in the cluster environment: balancing loads at the application level does not balance the process arrival patterns; and (3) the performance of collective communication algorithms is sensitive to process arrival patterns. These results indicate that process arrival pattern is an important factor that must be taken into consideration in developing and optimizing MPI collective routines. We propose a scheme that achieves high performance with different process arrival patterns, and demonstrate that by explicitly considering process arrival pattern, more efficient MPI collective routines than the current ones can be obtained.

show abstract

“…Our proposed scheme combines this idea with contention-free realization of reduce-scatter and all-gather operations. Various architecture specific allreduce schemes have also been developed [1,3,5,7,17]. In particular, all-reduce algorithms were developed specifically for SMP clusters in [17].…”

Section: Related Workmentioning

confidence: 99%