Identifying the Root Causes of Wait States in Large-Scale Parallel Applications

Abstract

Driven by growing application requirements and accelerated by current trends in microprocessor design, the number of processor cores on modern supercomputers is increasing from generation to generation. However, load or communication imbalance prevents many codes from taking advantage of the available parallelism, as delays of single processes may spread wait states across the entire machine. Moreover, when employing complex point-to-point communication patterns, wait states may propagate along far-reaching cause-effect chains that are hard to track manually and that complicate an assessment of the actual costs of an imbalance. Building on earlier work by Meira Jr. et al., we present a scalable approach that identifies program wait states and attributes their costs in terms of resource waste to their original cause. By replaying event traces in parallel both in forward and backward direction, we can identify the processes and call paths responsible for the most severe imbalances even for runs with tens of thousands of processes.

Large-Scale Performance Analysis of Sweep3d With the Scalasca Toolset

“…Key execution performance characteristics of Sweep3D were revealed by Scalasca runtime summarization and automated event trace analyses, and refined employing source code annotations inserted for major iteration loops and code sections to direct instrumentation and analysis. In on-going research we are investigating automatic determination and combining of iterations with similar performance profiles [17], and analyzing traces for the root causes of wait states to improve attribution of performance problems [18]. Tools for measuring and analyzing application execution performance also need to be highly scalable themself [19], as demonstrated by the Scalasca toolset with several hundred thousand Sweep3D processes on Cray XT5 and IBM BG/P, where multiple techniques for effective data reduction and management are employed and application-oriented graphical presentation facilitated insight into load-balance problems that only become critical at larger scales.…”

ScalOMP: Analyzing the Scalability of OpenMP Applications

“…Most of these works are focused on MPI; For example, performance models can help finding weak scaling issues [6]. A backward replay of an execution trace can be used for identifying the root cause of wait-states in MPI applications [5].…”

Performance analysis of concurrent workflows

“…In this context the classical monitoring approaches are combined with elements from complex event processing [8][9][10] in which the progress of a workflow is captured as an ordered set of events, the so-called trace or lifeline [11][12][13]. Applications comprise the identification of wait states in large scale simulations [14], the analysis of for-loops [15], or the visualization of execution and wait times in distributed systems [16].…”