A Generic Approach to Scheduling and Checkpointing Workflows

Han, Li; Fèvre, Valentin Le; Canon, Louis-Claude; Robert, Yves; Vivien, Frédéric

doi:10.1145/3225058.3225145

Cited by 11 publications

(9 citation statements)

References 40 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we compare our new general approach with P rop C kpt , the approach specific to M-SPGs that we proposed by Han et al (2018a). Figures 20 to 22 present this comparison for Montage, L igo and G enome , which are the three M-SPGs presented in the study by Han et al (2018a). Overall, the new approaches perform better than P rop C kpt .…”

Section: Methodsmentioning

confidence: 95%

See 1 more Smart Citation

A generic approach to scheduling and checkpointing workflows

Han

Fèvre

Canon

et al. 2019

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

This work deals with scheduling and checkpointing strategies to execute scientific workflows on failure-prone large-scale platforms. To the best of our knowledge, this work is the first to target fail-stop errors for arbitrary workflows. Most previous work addresses soft errors, which corrupt the task being executed by a processor but do not cause the entire memory of that processor to be lost, contrarily to fail-stop errors. We revisit classical mapping heuristics such as Heterogeneous Earliest Finish Time and MinMin and complement them with several checkpointing strategies. The objective is to derive an efficient trade-off between checkpointing every task (CkptAll), which is an overkill when failures are rare events, and checkpointing no task (CkptNone), which induces dramatic re-execution overhead even when only a few failures strike during execution. Contrarily to previous work, our approach applies to arbitrary workflows, not just special classes of dependence graphs such as minimal series-parallel graphs. Extensive experiments report significant gain over both CkptAll and CkptNone for a wide variety of workflows.

show abstract

Section: Methodsmentioning

confidence: 95%

“…Authors' Note A preliminary version of this work appeared in the proceedings of ACM ICPP 2018. A shorter version of this work has been published in the proceedings of ICPP'18 (Han et al, 2018b).…”

mentioning

confidence: 99%

A generic approach to scheduling and checkpointing workflows

Han

Fèvre

Canon

et al. 2019

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recall that #P is the class of counting problems that correspond to NP decision problems [28], and that #P-complete problems are at least as hard as NP-complete problems. Several heuristics to decide which tasks to checkpoint are proposed and evaluated in [14].…”

Section: Checkpointingmentioning

confidence: 99%

Robustness of the Young/Daly formula for stochastic iterative applications

Marchal

Pallez

et al. 2020

49th International Conference on Parallel Processing - ICPP

Self Cite

View full text Add to dashboard Cite

The Young/Daly formula for periodic checkpointing is known to hold for a divisible load application where one can checkpoint at any time-step. In an nutshell, the optimal period is P YD = 2µ f C where µ f is the Mean Time Between Failures (MTBF) and C is the checkpoint time. This paper assesses the accuracy of the formula for applications decomposed into computational iterations where: (i) the duration of an iteration is stochastic, i.e., obeys a probability distribution law D of mean µ D ; and (ii) one can checkpoint only at the end of an iteration. We first consider static strategies where checkpoints are taken after a given number of iterations k and provide a closed-form, asymptotically optimal, formula for k, valid for any distribution D. We then show that using the Young/Daly formula to compute k (as k•µ D = P YD ) is a first order approximation of this formula. We also consider dynamic strategies where one decides to checkpoint at the end of an iteration only if the total amount of work since the last checkpoint exceeds a threshold W th , and otherwise proceed to the next iteration. Similarly, we provide a closed-form formula for this threshold and show that P YD is a first-order approximation of W th . Finally, we provide an extensive set of simulations where D is either Uniform, Gamma or truncated Normal, which shows the global accuracy of the Young/Daly formula, even when the distribution D had a large standard deviation (and when one cannot use a first-order approximation). Hence we establish that the relevance of the formula goes well beyond its original framework.

show abstract

“…In some studies, in addition to fail‐stops, silent errors are considered . In , disk checkpointing is combined with memory checkpointing.…”

Section: Related Studiesmentioning

confidence: 99%

Combining replication and checkpointing redundancies for reducing resiliency overhead

Motallebi

2020

ETRI Journal

View full text Add to dashboard Cite

We herein propose a heuristic redundancy selection algorithm that combines resubmission, replication, and checkpointing redundancies to reduce the resiliency overhead in fault‐tolerant workflow scheduling. The appropriate combination of these redundancies for workflow tasks is obtained in two consecutive phases. First, to compute the replication vector (number of task replicas), we apportion the set of provisioned resources among concurrently executing tasks according to their needs. Subsequently, we obtain the optimal checkpointing interval for each task as a function of the number of replicas and characteristics of tasks and computational environment. We formulate the problem of obtaining the optimal checkpointing interval for replicated tasks in situations where checkpoint files can be exchanged among computational resources. The results of our simulation experiments, on both randomly generated workflow graphs and real‐world applications, demonstrated that both the proposed replication vector computation algorithm and the proposed checkpointing scheme reduced the resiliency overhead.

show abstract

A Generic Approach to Scheduling and Checkpointing Workflows

Cited by 11 publications

References 40 publications

A generic approach to scheduling and checkpointing workflows

A generic approach to scheduling and checkpointing workflows

Robustness of the Young/Daly formula for stochastic iterative applications

Combining replication and checkpointing redundancies for reducing resiliency overhead

Contact Info

Product

Resources

About