More Efficient Reduction Algorithms for Non-Power-of-Two Number of Processors in Message-Passing Parallel Systems

Rabenseifner, Rolf; Träff, Jesper Larsson

doi:10.1007/978-3-540-30218-6_13

Cited by 53 publications

(44 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many efficient platform independent algorithms for this operation have been proposed [22,23,25,27]. In [22], Rabenseifner…”

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

“…The butterfly-like algorithm has been developed some times ago [22,27] and has been extended to handle non-power-of-two numbers of processes [23]. Various architecture specific all-reduce schemes have also been developed [1,4,12,17,26].…”

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

“…We compare the proposed algorithms with native MPI implementations on these clusters. In addition, the traditional butterfly-like algorithm [22,23,27], denote as butterfly, that is theoretically both latency and bandwidth optimal when the network contention is not a problem is also compared. For high-end SMP/multi-core clusters, we also compare the proposed scheme with two SMP specific implementations.…”

Section: Methodsmentioning

confidence: 99%

“…At the end of the operation, all processes have all X-item results r 0 , r 1 , ..., r X−1 , where r j = a j 0 ⊕ a j 1 ⊕ ... ⊕ a j N −1 , 0 ≤ j ≤ X − 1. In the presence of rounding errors, MPI poses some quality requirements for this operation [23]. It is required that all processes must receive the same resulting data.…”

Section: All-reduce Operationmentioning

confidence: 99%

“…The results in all processors will be the same after the ring-based algorithm. Hence, the algorithm meets the minimum MPI quality requirement [23]. However, the bracketing for computing the results is not the same for different data items.…”

Section: Theoremmentioning

confidence: 99%

See 4 more Smart Citations

Bandwidth optimal all-reduce algorithms for clusters of workstations

Patarasuk

Yuan

2009

Journal of Parallel and Distributed Computing

300

147

View full text Add to dashboard Cite

We consider an efficient realization of the all-reduce operation with large data sizes in cluster environments, under the assumption that the reduce operator is associative and commutative. We derive a tight lower bound of the amount of data that must be communicated in order to complete this operation and propose a ring-based algorithm that only requires tree connectivity to achieve bandwidth optimality. Unlike the widely used butterfly-like all-reduce algorithm that incurs network contention in SMP/multi-core clusters, the proposed algorithm can achieve contention-free communication in almost all contemporary clusters including SMP/multi-core clusters and Ethernet switched clusters with multiple switches. We demonstrate that the proposed algorithm is more efficient than other algorithms on clusters with different nodal architectures and networking technologies when the data size is sufficiently large.

show abstract

“…Many efficient platform independent algorithms for this operation have been proposed [22,23,25,27]. In [22], Rabenseifner…”

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

Section: Ethernet Switched Cluster Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: All-reduce Operationmentioning

confidence: 99%

Section: Theoremmentioning

confidence: 99%

See 3 more Smart Citations

Bandwidth optimal all-reduce algorithms for clusters of workstations

Patarasuk

Yuan

2009

Journal of Parallel and Distributed Computing

300

147

View full text Add to dashboard Cite

show abstract

Towards performance portability through runtime adaptation for high‐performance computing applications

Gabriel¹,

Feki²,

Benkert³

et al. 2010

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYThe Abstract Data and Communication Library (ADCL) is an adaptive communication library optimizing application level group communication operations at runtime. The library provides for a given communication pattern a large number of implementations and incorporates a runtime selection logic in order to choose the implementation leading to the highest performance. In this paper, we demonstrate how an application utilizing ADCL is deployed on a wide range of HPC architectures, including an IBM Blue Gene/L, an NEC SX-8, an IBM Power PC cluster using an IBM Federation Switch, an AMD Opteron cluster utilizing a 4xInfiniBand and a Gigabit Ethernet network, and an Intel EM64T cluster using a hierarchical Gigabit Ethernet network with reduced uplink bandwidth. We demonstrate, how different implementations for the three-dimensional neighborhood communication lead to the minimal execution time of the application on different architectures. ADCL gives the user the advantage of having to maintain only a single version of the source code and still have the ability to achieve close to optimal performance for the application on all architectures.

show abstract

Non‐clairvoyant reduction algorithms for heterogeneous platforms

Benoît

Canon

Marchal

2014

Concurrency and Computation

View full text Add to dashboard Cite

Abstract:We revisit the classical problem of the reduction collective operation in a heterogeneous environment. We discuss and evaluate four algorithms that are non-clairvoyant, i.e., they do not know in advance the computation and communication costs. On the one hand, Binomial-stat and Fibonacci-stat are static algorithms that decide in advance which operations will be reduced, without adapting to the environment; they were originally defined for homogeneous settings. On the other hand, Tree-dyn and Non-Commut-Tree-dyn are fully dynamic algorithms, for commutative or non-commutative reductions. With identical computation costs, we show that these algorithms are approximation algorithms. When costs are exponentially distributed, we perform an analysis of Tree-dyn based on Markov chains. Finally, we assess the relative performance of all four non-clairvoyant algorithms with heterogeneous costs though a set of simulations.

show abstract

More Efficient Reduction Algorithms for Non-Power-of-Two Number of Processors in Message-Passing Parallel Systems

Cited by 53 publications

References 10 publications

Bandwidth optimal all-reduce algorithms for clusters of workstations

Bandwidth optimal all-reduce algorithms for clusters of workstations

Towards performance portability through runtime adaptation for high‐performance computing applications

Non‐clairvoyant reduction algorithms for heterogeneous platforms

Contact Info

Product

Resources

About