Automatically Tuned Collective Communications

Vadhiyar, Sathish; Fagg, Graham E.; Dongarra, Jack

doi:10.1109/sc.2000.10024

Cited by 98 publications

(83 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to other architecture specific collective communication algorithms [8,10,17], the techniques developed in this paper can be used in advanced communication libraries [7,9,13,30]. Our research extends the work in [11,23,28] by considering multiple switches. As shown in the performance study, pipelined broadcast using topology unaware trees in such an environment may yield extremely poor performance.…”

Section: Related Workmentioning

confidence: 76%

See 1 more Smart Citation

Techniques for pipelined broadcast on ethernet switched clusters

Patarasuk

Yuan

Faraj³

2008

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

By splitting a large broadcast message into segments and broadcasting the segments in a pipelined fashion, pipelined broadcast can achieve high performance in many systems. In this paper, we investigate techniques for efficient pipelined broadcast on clusters connected by multiple Ethernet switches. Specifically, we develop algorithms for computing various contention-free broadcast trees that are suitable for pipelined broadcast on Ethernet switched clusters, extend the parametrized LogP model for predicting appropriate segment sizes for pipelined broadcast, show that the segment sizes computed based on the model yield high performance, and evaluate various pipelined broadcast schemes through experimentation on Ethernet switched clusters with various topologies. The results demonstrate that our techniques are practical and efficient for contemporary fast Ethernet and Giga-bit Ethernet clusters.

show abstract

Section: Related Workmentioning

confidence: 76%

“…In [29], a pipelined broadcast technique is proposed for the mesh topology. The effectiveness of pipelined broadcast in cluster environments was demonstrated in [11,23,28]. It was shown that pipelined broadcast using topology unaware trees can be very efficient for clusters connected by a single switch.…”

Section: Related Workmentioning

confidence: 99%

Techniques for pipelined broadcast on ethernet switched clusters

Patarasuk

Yuan

Faraj³

2008

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…However, these works consider mostly homogeneous environments, and does not take the diversity of processor speed into considerationmostly because these approaches rely on recursive processor halving techniques and low level communication protocol improvements, and they may not perform well in heterogeneous clusters. Vadhiyar et al [33] gave an experimental approach to tune the collective communication, including reduction, via exhaustive search on heterogeneous clusters.…”

Section: Introductionmentioning

confidence: 99%

“…We want to emphasize that the exhaustive search techniques in this paper are guided by the theoretical foundation, and these techniques can be incorporated into the framework suggested by Vadhiyar et. al [33] that actually measures the key communication parameters, to further optimize the search efficiency.…”

Section: Introductionmentioning

confidence: 99%

An Approximation Algorithm and Dynamic Programming for Reduction in Heterogeneous Environments

2007

View full text Add to dashboard Cite

Network of workstation (NOW) is a cost-effective alternative to massively parallel supercomputers. As commercially available off-the-shelf processors become cheaper and faster, it is now possible to build a cluster that provides high computing power within a limited budget. However, a cluster may consist of different types of processors and this heterogeneity complicates the design of efficient collective communication protocols. For example, it is a very hard combinatorial problem to find an optimal reduction schedule for such heterogeneous clusters. Nevertheless, we show that a simple technique called slowest-node-first (SNF) is very effective in designing efficient reduction protocols for heterogeneous clusters. First, we show that SNF is actually a 2-approximation algorithm, which means that an SNF schedule length is always within twice of the optimal schedule length, no matter what kind of cluster is given. In addition, we show that SNF does give the optimal reduction time when the cluster consists of two types of processors, when the ratio of communication speed between them is at least two. When the communication speed ratio is less than two, we develop a dynamic programming technique to find the optimal schedule. Our dynamic programming utilizes the monotone property of the objective function, and can significantly reduce the amount of computation time. Finally, combined with an approximation algorithm for broadcast 2004, we propose an all-reduction algorithm which sends the reduction answer to all processors, with approximation ratio 3.5. We conduct three groups of experiments. First, we show that SNF performs better than the built-in MPI_Reduce in a test cluster. Second, we observe a factor of 93 times saving in computation time to find the optimal schedule, when compared with a naive dynamic programming implementation. Thirdly, we apply the theoretical results to a branch-and-bound search and show that they can reduce the search time of the optimal reduction schedule by a factor of 500, when the cluster has three kinds of processors.

show abstract

“…The current implementation is optimized for InfiniBand TM and implements different algorithms for most collective operations (cf. [15]). …”

Section: Implementation With Non-blocking Point-to-pointmentioning

confidence: 99%

A Case for Standard Non-blocking Collective Operations

Hoefler

Kambadur

Graham

et al.

Recent Advances in Parallel Virtual Machine and Message Passing Interface

View full text Add to dashboard Cite

Abstract. In this paper we make the case for adding standard nonblocking collective operations to the MPI standard. The non-blocking point-to-point and blocking collective operations currently defined by MPI provide important performance and abstraction benefits. To allow these benefits to be simultaneously realized, we present an application programming interface for non-blocking collective operations in MPI. Microbenchmark and application-based performance results demonstrate that non-blocking collective operations offer not only improved convenience, but improved performance as well, when compared to manual use of threads with blocking collectives.

show abstract

Automatically Tuned Collective Communications

Cited by 98 publications

References 7 publications

Techniques for pipelined broadcast on ethernet switched clusters

Techniques for pipelined broadcast on ethernet switched clusters

An Approximation Algorithm and Dynamic Programming for Reduction in Heterogeneous Environments

A Case for Standard Non-blocking Collective Operations

Contact Info

Product

Resources

About