High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62K processors

Abstract

SPECFEM3D_GLOBE is a spectral-element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knoxville, while a similar run on the 28K processor Jaguar system at ORNL, which has better memory bandwidth per processor, sustained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period.Thus we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe.We employed performance modeling methods to identify performance bottlenecks and worked through issues of parallel I/O and scalability. Improved mesh design and numbering results in excellent load balancing and few cache misses. The primary achievements are not just the scalability and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's interior at unprecedented resolution.

Scalable Graph Exploration on Multicore Processors

“…The insight provided by these two simple experiments is that, while a read-only access pattern can be easily scaled 2 lockb is the basic instruction to implement atomic operations on x86 processors, so the results can be directly generalized to other primitives such as read-and-or, that we use in the graph exploration. across multiple sockets relying on the native memory pipelining units, more sophisticated patterns that put pressure on the cache-coherency protocol require an innovative algorithmic solution.…”

“…This is a complex task because there are different types of locality: hierarchical networks, distributed, shared and local caches, local and non-local memory, and various cache-coherence effects. In an ideal spectrum of parallel applications, those that have high locality, rely on limited communication and closely match the cache hierarchy, can readily take advantage of existing multicore processors and accelerators [1], [2]. At the other end of the spectrum there are problems, such as graph exploration, with very little data reuse and a random access pattern.…”

Micro-applications for Communication Data Access Patterns and MPI Datatypes

“…Table 1 provides an overview of investigated application classes, their test cases, and a short description of the respective data access patterns. In detail, we analyzed the complex applications WRF [17], SPECFEM3D_GLOBE [7], MILC [4] and LAMMPS [15], representing the fields of weather simulation, seismic wave propagation, quantum chromodynamics and molecular dynamics. We also included existing parallel computing benchmarks and mini-apps, such as the NAS [19], the Sequoia benchmarks as well as the Mantevo mini apps [10].…”

“…It is used on some of the biggest HPC systems available [7]. Grid points that lie on the sides, edges or corners of an element are shared between neighboring elements.…”

Optimized viscoelastic wave propagation for weakly dissipative media

“…On the global scale, high-frequency body waves are routinely observed at propagation distances of 1000-2000 wavelengths and more, a regime that is hardly accessible with current global 3-D solvers and computers (Carrington et al 2008). While the methods we propose for parameter optimization are completely general and the coarse-grained memory variable approach applicable to all high-order finite-element methods, we use the axisymmetric SEM AxiSEM introduced by Nissen-Meyer et al (2007a,b, 2008, further developed to include anisotropy by and published open source by Nissen-Meyer et al (2014) as an example implementation to test our theoretical arguments.…”