Kinetic turbulence simulations at extreme scale on leadership-class systems

Abstract

Reliable predictive simulation capability addressing confinement properties in magnetically confined fusion plasmas is critically-important for ITER, a 20 billion dollar international burning plasma device under construction in France. The complex study of kinetic turbulence, which can severely limit the energy confinement and impact the economic viability of fusion systems, requires simulations at extreme scale for such an unprecedented device size. Our newly optimized, global, ab initio particle-in-cell code solving the nonlinear equations underlying gyrokinetic theory achieves excellent performance with respect to "time to solution" at the full capacity of the IBM Blue Gene/Q on 786,432 cores of Mira at ALCF and recently of the 1,572,864 cores of Sequoia at LLNL. Recent multithreading and domain decomposition optimizations in the new GTC-P code represent critically important software advances for modern, low memory per core systems by enabling routine simulations at unprecedented size (130 million grid points ITER-scale) and resolution (65 billion particles).

Extreme Scale Plasma Turbulence Simulations on Top Supercomputers Worldwide

“…Prior research involving large-scale PIC simulations, both standard PIC and gyrokinetic PIC, on leading supercomputer systems includes, for example, VPIC on Roadrunner [4], OSIRIS on Sequoia [5], PIConGPU on Titan [6], GTC-P on Mira and Sequoia [7], GTC on Tianhe-1A [8] and XGC1 on Titan [9]. The research carried out in this paper using the latest version of GTC-P-the fastest gyrokinetic PIC code worldwide with respect to time-to-solution of large problem size physics challenges-is the first to explore the portability, optimizations, and scalability across a wide range of multipetaflop platforms at the full or near-full capability, including both homogeneous and heterogeneous systems involving GPUs and Intel Xeon Phis.…”

“…As a result, particleand grid-based routines are distributed across all processes. We further exploit multicore parallelism using shared-memory multithreading which provides an additional multiplicative source of speedup [7].…”

“…Consequently, the message sizes in the toroidal dimension can be 10× larger than those in the radial dimension at each time step. On Blue Gene systems with explicit process mapping, we group processes to favor the MPI communicator in the toroidal dimension [7]. For other systems, assigning consecutive ranks for processes within each toroidal communicator generally leads to improved performance.…”

Experiences of Applying One-Sided Communication to Nearest-Neighbor Communication

“…The Princeton Gyrokinetic Toroidal Code (GTC-P) [33], [19] simulates plasma turbulence in magnetic confinement fusion devices called tokamaks. GTC-P follows the motion of ions and electrons in toroidal geometry by solving a 5D gyrophase-averaged Vlasov-Poisson equation using a particlein-cell (PIC) algorithm.…”

“…In this work, we study the performance of applying onesided communication on an adaptive mesh refinement framework (BoxLib) [2], and a scientific application GTC-P [33], [19], whose communication patterns are dominated by the nearest-neighbor communication. Both applications exhibit dynamic communication characteristics.…”