Large-Scale Performance Analysis of Sweep3d With the Scalasca Toolset

Abstract

Cray XT and IBM Blue Gene systems present current alternative approaches to constructing leadership computer systems relying on applications being able to exploit very large configurations of processor cores, and associated analysis tools must also scale commensurately to isolate and quantify performance issues that manifest at the largest scales. In studying the scalability of the Scalasca performance analysis toolset to several hundred thousand MPI processes on XT5 and BG/P systems, we investigated a progressive execution performance deterioration of the well-known ASCI Sweep3D compact application. Scalasca runtime summarization analysis quantified MPI communication time that correlated with computational imbalance, and automated trace analysis confirmed growing amounts of MPI waiting times. Further instrumentation, measurement and analyses pinpointed a conditional section of highly imbalanced computation which amplified waiting times inherent in the associated wavefront communication that seriously degraded overall execution efficiency at very large scales. By employing effective data collation, management and graphical presentation, in a portable and straightforward to use toolset, Scalasca was thereby able to demonstrate performance measurements and analyses with 294,912 processes.

Using automated performance modeling to find scalability bugs in complex codes

“…On a final note, this relationship sheds light also on a performance phenomenon observed in a more recent experimental study of Sweep3D [38], which analyzes the consequences of load imbalance between a central rectangular region and the rest of the process grid, which is caused by a corrective function invoked only during certain iterations. Since overload has effects similar to processors with lower speed, it is likely to enlarge only the coefficient of the √ p term in the model of the dominant receives and, thus, to have only little bearing on the general scalability.…”

Separating the wheat from the chaff

“…(2) The second similarity group in the line-feature class, which is shown in 4% of the performance views, exposes a visual structure consisting of oblique lines radiating from the interior to the border of the domain. As described by Wylie et al in a Sweep3D scaling study [18], these oblique lines reveal a computational imbalance that is due to so called "fixup iterations", which apply corrections to negative fluxes and are necessary for a physically realistic solution. (3) The third similarity group in the linefeature class, which is shown in 8% of the performance views, exposes a visual structure consisting of long vertical lines.…”

“…In this section we describe typical challenges in exploring performance data that need to be considered for defining a visual-analytics driven analysis workflow. Table 1 shows the sizes of performance-data search spaces from three applications examples (namely Sweep3D [17,18], CICE [19,20] and PFLOTRAN [21,22]). Taking into account the number of the performance metrics, call paths and system resources together with its tree structures and in-and exclusive entities, the number of available data items is in the order of a billion.…”

“…For the parallelization, the originally 3D computational domain is mapped onto a 2D virtual topology. The performance data set used is a profile created through an eventtrace analysis with Scalasca [18]. The experiment was conducted with 294,912 MPI processes on an IBM Blue Gene/P system at the Jülich Supercomputing Centre.…”

“…(4) The one similarity group in the edge-feature class, which is shown in 8% of the performance views, exposes a visual structure consisting of a central rectangle. The central rectangle reveals a load imbalance and is, like the oblique lines of similarity group (2), also the result of the "fixup iterations" [18]. (5) The one similarity group in the gradient-feature class, which is shown in 2% of the performance views, exposes a visual structure consisting of a diagonal smooth gradient.…”