The Role of Idle Waves, Desynchronization, and Bottleneck Evasion in the Performance of Parallel Programs

Abstract

The performance of highly parallel applications on distributed-memory systems is influenced by many factors. Analytic performance modeling techniques aim to provide insight into performance limitations and are often the starting point of optimization efforts. However, coupling analytic models across the system hierarchy (socket, node, network) fails to encompass the intricate interplay between the program code and the hardware, especially when execution and communication bottlenecks are involved. In this paper we investigate the effect of bottleneck evasion and how it can lead to automatic overlap of communication overhead with computation. Bottleneck evasion leads to a gradual loss of the initial bulk-synchronous behavior of a parallel code so that its processes become desynchronized. This occurs most prominently in memory-bound programs, which is why we choose memory-bound benchmark and application codes, specifically an MPI-augmented STREAM Triad, sparse matrix-vector multiplication, and a collective-avoiding Chebyshev filter diagonalization code to demonstrate the consequences of desynchronization on two different supercomputing platforms. We investigate the role of idle waves as possible triggers for desynchronization and show the impact of automatic asynchronous communication for a spectrum of code properties and parameters, such as saturation point, matrix structures, domain decomposition, and communication concurrency. Our findings reveal how eliminating synchronization points (such as collective communication or barriers) precipitates performance improvements that go beyond what can be expected by simply subtracting the overhead of the collective from the overall runtime.

Exploring Techniques for the Analysis of Spontaneous Asynchronicity in MPI-Parallel Applications

“…The histograms in Figure 3 sort the MPI time values of end-to-end (500 k iterations) runs of Bench [1][2][3][4]A into 35 bins. For memory-bound code, idle times are lower for desynchronized processes if the bandwidth saturation on a ccNUMA domain is weaker [2] (see Figure 3(c)).…”

“…As a consequence, such programs settle in a metastable state, a computational wavefront, where neighboring processes are shifted in time with respect to each other (Figure 2 (right)). It was also shown [4] that this desynchronization can lead to substantial speedups via automatic overlap of communication and code execution.…”