Numerical recovery strategies for parallel resilient Krylov linear solvers

Abstract

International audienceAs the computational power of high performance computing (HPC) systems continues to increase by using a huge number of cores or specialized processing units, HPC applications are increasingly prone to faults. In this paper, we present a new class of numerical fault tolerance algorithms to cope with node crashes in parallel distributed environments. This new resilient scheme is designed at application level and does not require extra resources, i.e., computational unit or computing time, when no fault occurs. In the framework of iterative methods for the solution of sparse linear systems, we present numerical algorithms to extract relevant information from available data after a fault, assuming a separate mechanism ensures the fault detection. After data extraction, a well chosen part of missing data is regenerated through interpolation strategies to constitute meaningful inputs to restart the iterative scheme. We have developed these methods, referred to as Interpolation-Restart techniques, for Krylov subspace linear solvers. After a fault, lost entries of the current iterate computed by the solver are interpolated to define a new initial guess to restart the Krylov method. A well suited initial guess is computed by using the entries of the faulty iterate available on surviving nodes. We present two interpolation policies that preserve key numerical properties of well-known linear solvers, namely the monotonic decrease of the A-norm of the error of the conjugate gradient or the residual norm decrease of GMRES. The qualitative numerical behavior of the resulting scheme have been validated with sequential simulations, when the number of faults and the amount of data losses are varied. Finally, the computational costs associated with the recovery mechanism have been evaluated through parallel experiments

How to Make the Preconditioned Conjugate Gradient Method Resilient Against Multiple Node Failures

“…1.1.3 Notation. We use a notation similar to [2] to denote sections of matrices and vectors. We refer to the set of all indices as I {1, 2, .…”

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction

“…Interpolation-Restart (IR) techniques are designed to cope with node crashes (hard faults) in a parallel distributed environment (Agullo et al, 2015(Agullo et al, , 2017(Agullo et al, , 2016a(Agullo et al, , 2016b. The methods can be designed at the algebraic level for the solution both of linear systems and of eigenvalue problems.…”

Algorithm-based fault recovery of adaptively refined parallel multilevel grids

“…For sparse matrix iterative solvers, such as SOR, GMRES and CG-iterations, the previously mentioned approaches are not readily applicable (Sloan et al, 2012). Suitable extensions are presented in (Agullo et al, 2013; Bridges et al, 2012; Chen, 2013; Roy-Chowdhury and Banerjee, 1993; Stoyanov and Webster, 2013). Cui et al (2017) exploit the mathematical structure of a parallel subspace correction method.…”