Application-Level Differential Checkpointing for HPC Applications with Dynamic Datasets

Keller, Kai; Bautista-Gomez, Leonardo

doi:10.1109/ccgrid.2019.00015

Cited by 8 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Resilient checkpointing has been considered with the help of nonvolatile memory, as for instance implemented in PapyrusKV (Kim et al, 2017), a resilient key-value blob-storage. Other resilient checkpointing techniques include the self-checkpoint technique (Tang et al, 2018), which reduces common redundancies while writing checkpoints, or techniques reducing the amount of required memory through hierarchical checkpointing (Moody et al, 2010), or differential checkpointing (Keller and Bautista-Gomez, 2019).…”

Section: System Infrastructure Techniques For Resiliencementioning

confidence: 99%

Resiliency in numerical algorithm design for extreme scale simulations

Agullo

Altenbernd

Anzt

et al. 2021

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

This work is based on the seminar titled ‘Resiliency in Numerical Algorithm Design for Extreme Scale Simulations’ held March 1–6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 1023 floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.

show abstract

Section: System Infrastructure Techniques For Resiliencementioning

confidence: 99%

Resiliency in numerical algorithm design for extreme scale simulations

Agullo

Altenbernd

Anzt

et al. 2021

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

show abstract

“…In essence during the checkpoint procedure one needs to store only the data that have changed their value from the previous checkpoint. This technique is called differential checkpoint [18], and can be implemented in two ways. The first one, uses the page dirty bits 2 of the Operating System (OS) to detect which pages has changed from the previous checkpoint.…”

Section: Differential Checkpointmentioning

confidence: 99%

“…A number of optimizations have been proposed to reduce the amount of data to be stored. Differential approaches [10], [11], [18] update only the data that has changed in comparison to the previous checkpoint and are beneficial only when applications do not change substantially. In contrast to these works, our work focuses on the programmability aspect of application initiated multilevel C/R and optimizes the normal checkpoint procedure.…”

Section: Related Workmentioning

confidence: 99%

Checkpoint Restart Support for Heterogeneous HPC Applications

Parasyris

Keller

Bautista-Gomez

et al. 2020

2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID)

Self Cite

View full text Add to dashboard Cite

As we approach the era of exa-scale computing, fault tolerance is of growing importance. The increasing number of cores as well as the increased complexity of modern heterogenous systems result in substantial decrease of the expected mean time between failures. Among the different fault tolerance techniques, checkpoint/restart is vastly adopted in supercomputing systems. Although many supercomputers in the TOP 500 list use GPUs, only a few checkpoint restart mechanism support GPUs. In this paper, we extend an application level checkpoint library, called fault tolerance interface (FTI), to support multinode/multi-GPU checkpoints. In contrast to previous work, our library includes a memory manager, which upon a checkpoint invocation tracks the actual location of the data to be stored and handles the data accordingly. We analyze the overhead of the checkpoint/restart procedure and we present a series of optimization steps to massively decrease the checkpoint and recovery time of our implementation. To further reduce the checkpoint time we present a differential checkpoint approach which writes only the updated data to the checkpoint file. Our approach is evaluated and, in the best case scenario, the execution time of a normal checkpoint is reduced by 15x in contrast with a non-optimized version, in the case of differential checkpoint the overhead can drop to 2.6% when checkpointing every 30s.

show abstract

“…Hasta ahora, las soluciones que hemos aportado en este trabajo se basan en la utilización de checkpoints coordinados de nivel de sistema (DM T CP ) o no-coordinados (por proceso, en capa de aplicación). Sin embargo, existen también otras variantes, como los checkpoints semicoordinados o el checkpointing diferencial (un método que reduce la carga de Entrada/Salida de realizar checkpoints consecutivos actualizando sólo aquellos bloques de datos que han cambiado desde el almacenamiento del último checkpoint [75]), que no hemos tenido en cuenta hasta el momento.…”

Section: Trabajos Futurosunclassified

SEDAR: Detección y recuperación automática de fallos transitorios en sistemas de cómputo de altas prestaciones

Montezanti¹

View full text Add to dashboard Cite

El manejo de fallos es una preocupación creciente en el contexto del HPC; en el futuro, se esperan mayores variedades y tasas de errores, intervalos de detección más largos y fallos silenciosos. Se proyecta que, en los próximos sistemas de exa-escala, los errores ocurran incluso varias veces al día y se propaguen en grandes aplicaciones paralelas, generando desde caídas de procesos hasta corrupciones de resultados debidas a fallos no detectados. En este trabajo se propone SEDAR, una metodología que mejora la fiabilidad, frente a los fallos transitorios, de un sistema que ejecuta aplicaciones paralelas de paso de mensajes. La solución diseñada, basada en replicación de procesos para la detección, combinada con diferentes niveles de checkpointing (checkpoints de nivel de sistema o de nivel de aplicación) para recuperar automáticamente, tiene el objetivo de ayudar a los usuarios de aplicaciones científicas a obtener ejecuciones confiables con resultados correctos. La detección se logra replicando internamente cada proceso de la aplicación en threads y monitorizando los contenidos de los mensajes entre los threads antes de enviar a otro proceso; además, los resultados finales se validan para prevenir la corrupción del cómputo local. Esta estrategia permite relanzar la ejecución desde el comienzo ni bien se produce la detección, sin esperar innecesariamente hasta la conclusión incorrecta. Para la recuperación, se utilizan checkpoints de nivel de sistema, pero debido a que no existe garantía de que un checkpoint particular no contenga errores silenciosos latentes, se requiere el almacenamiento y mantenimiento de múltiples checkpoints, y se implementa un mecanismo para reintentar recuperaciones sucesivas desde checkpoints previos si el mismo error se detecta nuevamente. La última opción es utilizar un único checkpoint de capa de aplicación, que puede ser verificado para asegurar su validez como punto de recuperación seguro. En consecuencia, SEDAR se estructura en tres niveles: (1) sólo detección y parada segura con notificación al usuario; (2) recuperación basada en una cadena de checkpoints de nivel de sistema; y (3) recuperación basada en un único checkpoint válido de capa de aplicación. Cada una de estas variantes brinda una cobertura particular, pero tiene limitaciones inherentes y costos propios de implementación; la posibilidad de elegir entre ellos provee flexibilidad para adaptar la relación costo-beneficio a las necesidades de un sistema particular. Se presenta una descripción completa de la metodología, su comportamiento en presencia de fallos y los overheads temporales de emplear cada una de las alternativas. Se describe un modelo que considera varios escenarios de fallos y sus efectos predecibles sobre una aplicación de prueba para realizar una verificación funcional. Además, se lleva a cabo una validación experimental sobre una implementación real de la herramienta SEDAR, utilizando diferentes benchmarks con patrones de comunicación disímiles. El comportamiento en presencia de fallos, inyectados controladamente en distintos momentos de la ejecución, permite evaluar el desempeño y caracterizar el overhead asociado a su utilización. Tomando en cuenta esto, también se establecen las condiciones bajo las cuales vale la pena comenzar con la protección y almacenar varios checkpoints para recuperar, en lugar de simplemente detectar, detener la ejecución y relanzar. Las posibilidades de configurar el modo de uso, adaptándolo a los requerimientos de cobertura y máximo overhead permitido de un sistema particular, muestran que SEDAR es una metodología eficaz y viable para la tolerancia a fallos transitorios en entornos de HPC.

show abstract

Application-Level Differential Checkpointing for HPC Applications with Dynamic Datasets

Cited by 8 publications

References 14 publications

Resiliency in numerical algorithm design for extreme scale simulations

Resiliency in numerical algorithm design for extreme scale simulations

Checkpoint Restart Support for Heterogeneous HPC Applications

SEDAR: Detección y recuperación automática de fallos transitorios en sistemas de cómputo de altas prestaciones

Contact Info

Product

Resources

About