Fault-Tolerant Grid-Based Solvers: Combining Concepts from Sparse Grids and MapReduce

Larson, Jay; Hegland, Markus; Harding, Brendan; Roberts, Stephen; Stals, Linda; Rendell, Alistair P.; Strazdins, Peter; Ali, Md. Mohsin; Kowitz, Christoph; Nobes, Ross H.; Southern, James; Wilson, Nicholas E.; Li, M.; Oishi, Y.

doi:10.1016/j.procs.2013.05.176

Cited by 16 publications

(10 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chen et al [15] proposed an algorithm-based recovery method for iterative system solvers to enable resilience to fail-stop failures based on data partitioning tailored to the characteristics of the iterative scheme, while Larson et al [30] achieved fault tolerance by combining solutions on sparse grids. Both approaches can in principle be reformulated in a domain decomposition paradigm, but with a distinct flavor of fault-detection or redundancy present.…”

mentioning

confidence: 99%

Fault Resilient Domain Decomposition Preconditioner for PDEs

Sargsyan¹,

Rizzi²,

Mycek

et al. 2015

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

The move towards extreme-scale computing platforms challenges scientific simulations in many ways. Given the recent tendencies in computer architecture development, one needs to reformulate legacy codes in order to cope with large amounts of communication, system faults and requirements of low-memory usage per core. In this work, we develop a novel framework for solving partial differential equations (PDEs) via domain decomposition that reformulates the solution as a state-of-knowledge with a probabilistic interpretation. Such reformulation allows resiliency with respect to potential faults without having to apply fault detection, avoids unnecessary communication and is generally well-suited for rigorous uncertainty quantification studies that target improvements of predictive fidelity of scientific models. We demonstrate our algorithm for one-dimensional PDE examples where artificial faults have been implemented as bit-flips in the binary representation of subdomain solutions.

show abstract

mentioning

confidence: 99%

Fault Resilient Domain Decomposition Preconditioner for PDEs

Sargsyan¹,

Rizzi²,

Mycek

et al. 2015

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

show abstract

“…As in many other applications, we use Python to glue together the different components of our implementation as well as providing some high level functions for performing the combination. It is based upon the development of NuMRF [17,18]:…”

Section: Top Layer: Loadmentioning

confidence: 99%

“…The second part is the initialisation of data structures and construction/allocation of arrays which will hold the relevant data. This is achieved using PyGraFT [17,18] which is a general class of grids and fields that allows us to handle data from the various components in a generic way. Also in this part of the code is the building of a sparse grid data structure.…”

Section: Top Layer: Loadmentioning

confidence: 99%

Fault Tolerant Computation with the Sparse Grid Combination Technique

Harding¹,

Hegland²,

Larson³

et al. 2015

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

This paper continues to develop a fault tolerant extension of the sparse grid combination technique recently proposed in [B. Harding and M. Hegland, ANZIAM J. Electron. Suppl., 54 (2013), pp. C394-C411]. This approach to fault tolerance is novel for two reasons: First, the combination technique adds an additional level of parallelism, and second, it provides algorithmbased fault tolerance so that solutions can still be recovered if failures occur during computation. Previous work indicates how the combination technique may be adapted for a low number of faults. In this paper we develop a generalization of the combination technique for which arbitrary collections of coarse approximations may be combined to obtain an accurate approximation. A general fault tolerant combination technique for large numbers of faults is a natural consequence of this work. Using a renewal model for the time between faults on each node of a high performance computer, we also provide bounds on the expected error for interpolation with this algorithm in the presence of faults. Numerical experiments solving the scalar advection PDE demonstrate that the algorithm is resilient to faults on a real application. It is observed that the time to solution is not significantly affected by the presence of (simulated) faults. Additionally the expected error increases with the number of faults but is relatively small even for high fault rates. A comparison with traditional checkpoint-restart methods applied to the combination technique shows that our approach is highly scalable with respect to the number of faults.

show abstract

“…Hadoop is widely used in large-scale data analysis in the commercial and research communities. The rapid uptake of MapReduce has spawned other implementations, including one utilizing Python's Pool class [21], and the message-passing-parallel MapReduce-MPI [22]. A review discussion of MapReduce implementations, advantages and disadvantages of the MapReduce programming model, and its variants and extensions is given by [23].…”

Section: Mapreducementioning

confidence: 99%

“…Algorithm-Based Fault Tolerance (ABFT) techniques for creating robust PDE solvers based on the modified sparse grid combination technique are proposed in [22], [9]. The proposed solver can accommodate the loss of single or multiple component grids.…”

Section: Related Workmentioning

confidence: 99%

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Ali

Strazdins

Harding

et al. 2015

2015 International Conference on High Performance Computing &Amp; Simulation (HPCS)

View full text Add to dashboard Cite

Applications performing ultra-large scale simulations via solving PDEs require very large computational systems for their timely solution. Studies have shown the rate of failure grows with the system size and these trends are likely to worsen in future machines as less reliable components are used to reduce the energy cost. Thus, as systems, and the problems solved on them, continue to grow, the ability to survive failures is becoming a critical aspect of algorithm development. The sparse grid combination technique (SGCT) is a cost-effective method for solving time-evolving PDEs, especially for higherdimensional problems. It can also be easily modified to provide algorithm-based fault tolerance for these problems. In this paper, we show how the SGCT can produce a fault-tolerant version of the GENE gyrokinetic plasma application, which evolves a 5D complex density field over time. We use an alternate component grid combination formula to recover data from lost processes. User Level Failure Mitigation (ULFM) MPI is used to recover the processes, and our implementation is robust over multiple failures and recovery for both process and node failures. An acceptable degree of modification of the application is required. Results using the SGCT on two of the fields' dimensions show competitive execution times with acceptable error (within 0.1%), compared to the same simulation with a single full resolution grid. The benefits improve when the SGCT is used over three dimensions. Our experiments show that the GENE application can successfully recover from multiple process failures, and applying the SGCT the corresponding number of times minimizes the error for the lost sub-grids. Application recovery overhead via ULFM MPI increases from ∼1.5s at 64 cores to ∼5s at 2048 cores for a oneoff failure. This compares favourably to using GENE's in-built checkpointing with job restart in conjunction with the classical SGCT on failure, which have overheads four times as large for a single failure, excluding the backtrack overhead. An analysis for a long-running application taking into account checkpoint backtrack times indicates a reduction in overhead of over an order of magnitude.

show abstract

Fault-Tolerant Grid-Based Solvers: Combining Concepts from Sparse Grids and MapReduce

Cited by 16 publications

References 24 publications

Fault Resilient Domain Decomposition Preconditioner for PDEs

Fault Resilient Domain Decomposition Preconditioner for PDEs

Fault Tolerant Computation with the Sparse Grid Combination Technique

A fault-tolerant gyrokinetic plasma application using the sparse grid combination technique

Contact Info

Product

Resources

About