Leveraging non-blocking collective communication in high-performance applications

Hoefler, Torsten; Gottschling, Peter; Lumsdaine, Andrew

doi:10.1145/1378533.1378554

Cited by 20 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This overlap potential of computation and communication has been analyzed by Hoefler et al and Hoefler and Lumsdaine [13,14]. However, nonblocking collective operations also provide semantic advantages that allow separate start and completion of a globally synchronizing operation.…”

Section: Algorithm N Bx -Nonblocking Consensusmentioning

confidence: 99%

Scalable communication protocols for dynamic sparse data exchange

Hoefler

Siebert²,

Lumsdaine

2010

Proceedings of the 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

Self Cite

View full text Add to dashboard Cite

Many large-scale parallel programs follow a bulk synchronous parallel (BSP) structure with distinct computation and communication phases. Although the communication phase in such programs may involve all (or large numbers) of the participating processes, the actual communication operations are usually sparse in nature. As a result, communication phases are typically expressed explicitly using point-to-point communication operations or collective operations. We define the dynamic sparse data-exchange (DSDE) problem and derive bounds in the well known LogGP model. While current approaches work well with static applications, they run into limitations as modern applications grow in scale, and as the problems that are being solved become increasingly irregular and dynamic. To enable the compact and efficient expression of the communication phase, we develop suitable sparse communication protocols for irregular applications at large scale. We discuss different irregular applications and show the sparsity in the communication for real-world input data. We discuss the time and memory complexity of commonly used protocols for the DSDE problem and develop N BX -a novel fast algorithm with constant memory overhead for solving it. Algorithm N BX improves the runtime of a sparse dataexchange among 8,192 processors on BlueGene/P by a factor of 5.6. In an application study, we show improvements of up to a factor of 28.9 for a parallel breadth first search on 8,192 BlueGene/P processors.

show abstract

Section: Algorithm N Bx -Nonblocking Consensusmentioning

confidence: 99%

Scalable communication protocols for dynamic sparse data exchange

Hoefler

Siebert²,

Lumsdaine

2010

Proceedings of the 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although the communication volumes of IS and FT are not as high as CG's, their bulky communication behavior based mostly on all-to-all type operations causes problems for low-bandwidth environments. Recent work by Hoefler et al [21] on non-blocking collective communication operations demonstrated benefits for Fast-FourierTransform operations, such as those used in FT, by allowing collective communication to overlap with computation. A version of FT using nonblocking collective could therefore extend the range of execution environments that lead to reasonable performance.…”

Section: Discussion On Suitable Applications For Parallel Volunteer Cmentioning

confidence: 99%

A Robust and Efficient Message Passing Library for Volunteer Computing Environments

et al. 2010

View full text Add to dashboard Cite

The objective of this research is to convert ordinary idle PCs into virtual clusters for executing parallel applications. The paper presents VolpexMPI that is designed to enable seamless forward application progress in the presence of frequent node failures as well as dynamically changing networks and node execution speeds. Process replication is employed to provide robustness. The central challenge in the design of VolpexMPI is to efficiently and automatically manage dynamically varying number of process replicas in different states of execution progress.The key fault tolerance technique employed is fully distributed sender based logging. The paper presents the design and an implementation of VolpexMPI. Preliminary results validate that the overhead of providing robustness is modest for applications with a favorable ratio of communication to computation and a low degree of communication.

show abstract

“…Original HPL chain broadcast hide latency costs, properly applying them to existing realworld applications is non-trivial. Their use often requires significant restructuring to exploit communication/computation overlap fully [20]. This requirement confronts the programmer with yet more complexity in the optimization process.…”

Section: Automatically Updating To Mpi 30mentioning

confidence: 99%

Exploitation of Dynamic Communication Patterns through Static Analysis

Preissl

Supinski

Schulz

et al. 2010

2010 39th International Conference on Parallel Processing

View full text Add to dashboard Cite

Abstract-Collective operations can have a large impact on the performance of parallel applications. However, the ideal implementation of a particular collective communication often depends on both the application and the targeted machine structure. Our approach combines dynamic and static analysis techniques to identify common collective communication patterns expressed as point-to-point calls and transforms them into equivalent MPI collectives. We first detect potential collective communication patterns in runtime traces and associate them with the corresponding source code regions. If our static analysis verifies that the introduction of collectives is safe for any program flow, we then replace the original communication primitives with their collective counterpart. In this paper we introduce the necessary algorithms to determine the safety of these transformations and we demonstrate several use cases, including automatic use of new extensions to the MPI standard such as nonblocking collective operations. The use of dynamic analysis significantly reduces compile times, resulting in a speed-up of about 50 for source transformations of HPL due to more directed analysis capabilities and also dramatically decreases complexity of the underlying static analysis.

show abstract

Leveraging non-blocking collective communication in high-performance applications

Cited by 20 publications

References 10 publications

Scalable communication protocols for dynamic sparse data exchange

Scalable communication protocols for dynamic sparse data exchange

A Robust and Efficient Message Passing Library for Volunteer Computing Environments

Exploitation of Dynamic Communication Patterns through Static Analysis

Contact Info

Product

Resources

About