Efficient checkpoint/verification patterns

Abstract

International audienceErrors have become a critical problem for high performance computing. Checkpointing protocols are often used for error recovery after fail-stop failures. However, silent errors cannot be ignored, and their peculiarity is that such errors are identified only when the corrupted data is activated. To cope with silent errors, we need a verification mechanism to check whether the application state is correct. Checkpoints should be supplemented with verifications to detect silent errors. When a verification is successful, only the last checkpoint needs to be kept in memory because it is known to be correct. In this paper, we analytically determine the best balance of verifications and checkpoints so as to optimize platform throughput. We introduce a balanced algorithm using a pattern with p checkpoints and q verifica-tions, which regularly interleaves both checkpoints and verifications across same-size computational chunks. We show how to compute the waste of an arbitrary pattern, and we prove that the balanced algorithm is optimal when the platform MTBF (Mean Time Between Failures) is large in front of the other parameters (checkpointing, verification and recovery costs). We conduct several simulations to show the gain achieved by this balanced algorithm for well-chosen values of p and q, compared to the base algorithm that always perform a verification just before taking a checkpoint (p = q = 1), and we exhibit gains of up to 19%

Assessing the Impact of Partial Verifications against Silent Data Corruptions

“…Furthermore, we determine the optimal configuration of a partial verification when its cost and recall can be traded off with each other. These results provide important extensions to the classical formulas in the field [27], [11], [4], [3], and to the best of our knowledge, are the first to include partial verifications. Unlike in the classical case, however, a silent error may not be detected by a partial verification and could get propagated to the subsequent work segments inside a pattern, thus significantly complicating the analysis.…”

“…For example, in the classical protocol for fail-stop errors where verification is not needed, the optimal checkpointing period is known to be √ 2μC as given by Young [27] and Daly [11]. A similar result is also known for silent errors, and the optimal period in that case is μ(C + V * ) if only verified checkpoints are used [4], [3]. These formulas provide firstorder approximations to the optimal patterns in the respective scenarios, and are valid when the resilient parameters satisfy C, V * μ.…”

“…The latter pattern, which needs to maintain only one checkpoint, is also analyzed in [3] to accommodate both fail-stop and silent errors. Benoit, Raina and Robert [4] extend the analysis of [1] by including p checkpoints and q verifications that are interleaved to form arbitrary patterns. All of these results assume the use of guaranteed verifications only.…”

“…This assumption is commonly made in the literature (see, e.g., [27], [11], [1], [3], [4]), which allows to make first-order approximations in the analysis to obtain close-form solutions.…”

“…This simple protocol eliminates the drawbacks of the multiple-checkpoint approach, provided that a guaranteed verification mechanism can be efficiently implemented. Of course, one can also design more sophisticated protocols by coupling multiple verifications with one checkpoint or even interleaving multiple checkpoints and verifications [1], [4]. The optimal pattern (i.e., number of verifications per checkpoint) in these protocols would be determined by the relative cost of executing a verification compared to checkpointing.…”