A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Abstract

Applications performing ultra-large scale simulations via solving PDEs require very large computational systems for their timely solution. Studies have shown the rate of failure grows with the system size and these trends are likely to worsen in future machines as less reliable components are used to reduce the energy cost. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) is a cost-effective method for solving time-evolving PDEs, especially for higherdimensional problems. It can also be easily modified to provide algorithm-based fault tolerance for these problems. In this paper, we show how the SGCT can produce a fault-tolerant version of the GENE gyrokinetic plasma application, which evolves a 5D complex density field over time. We use an alternate component grid combination formula to recover data from lost processes. User Level Failure Mitigation (ULFM) MPI is used to recover the processes, and our implementation is robust over multiple failures and recovery for both process and node failures. An acceptable degree of modification of the application is required. Results using the SGCT on two of the fields' dimensions show competitive execution times with acceptable error (within 0.1%), compared to the same simulation with a single full resolution grid. The benefits improve when the SGCT is used over three dimensions. Our experiments show that the GENE application can successfully recover from multiple process failures, and applying the SGCT the corresponding number of times minimizes the error for the lost sub-grids. Application recovery overhead via ULFM MPI increases from ∼1.5s at 64 cores to ∼5s at 2048 cores for a oneoff failure. This compares favourably to using GENE's in-built checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the backtrack overhead. An analysis for a long-running application taking into account checkpoint backtrack times indicates a reduction in overhead of over an order of magnitude.

Sparse grid techniques for particle-in-cell schemes

“…The idea of a sparse grid is not new, having been studied extensively in the applied mathematics community [8,21,24,25]. It has also seen some use in continuum codes for plasma applications [2,26,27,[29][30][31]37].…”

Application of a Binary Filter Inspired from the Pic Sparse Grid Technique to the Xtor-K Code

“…The resulting numerical solution is significantly denoised with a moderate increase in numerical complexity. Although the sparse grid combination technique had already been used in plasma physics 16,17 , it was first suggested in reference 1 as a way to reduce the PIC noise. Following this seminal paper, the merits of the sparse grid PIC technique have been exploited in plasma applications pertaining to radio frequency discharges 18 and to the electron drift instability in Hall thrusters 19 .…”

Resiliency in numerical algorithm design for extreme scale simulations

“…differently discretized) grids anyway, we can use these to detect anomalies – just by comparing the available solutions. And the detection leads immediately to a mitigation strategy (see Section 3.2.2), since we can easily exchange coarse grids in case of errors, just by changing the combination pattern (Ali et al, 2015; Ali et al, 2016; Harding & et al, 2015; Heene et al, 2016; Hinojosa et al, 2016; Obersteiner et al, 2017). Therefore, this is an example for a smart algorithm that is able to do both detection and mitigation.…”