Fault Tolerance and Recovery of Scientific Workflows on Computational Grids

Abstract

Abstract-In this paper, we describe the design and implementation of two mechanisms for fault-tolerance and recovery for complex scientific workflows on computational grids. We present our algorithms for over-provisioning and migration, which are our primary strategies for fault-tolerance. We consider application performance models, resource reliability models, network latency and bandwidth and queue wait times for batch-queues on compute resources for determining the correct fault-tolerance strategy. Our goal is to balance reliability and performance in the presence of soft, real-time constraints like deadlines and expected success probabilities, and to do it in a way that is transparent to scientists. We have evaluated our strategies by developing a Fault-Tolerance and Recovery (FTR) service and deploying it as a part of the Linked Environments for Atmospheric Discovery (LEAD) production infrastructure. Results from real usage scenarios in LEAD show that the failure rate of individual steps in workflows decreases from about 30% to 5% by using our fault-tolerance strategies.

Combined Fault Tolerance and Scheduling Techniques for Workflow Applications on Computational Grids

“…Integrating Fault Tolerance with HEFT/DSH: Figure 1 describes the algorithm that we use to integrate the over-provisioning algorithm in [11] with HEFT. First, we sort the tasks in the DAG by its upward rank.…”

“…Our goal is to find the smallest set of resources to replicate the given workflow task to satisfy these constraints. We use an efficient algorithm described by Kandaswamy et al [11] to find the smallest subset of resources that satisfies these constraints. In the cases when it is not possible to satisfy the success probability or deadline constraints, the over-provisioning algorithm returns all possible resource combinations tagged with the success probabilities for each resource set, so that a best-effort replicated set of resources can be chosen.…”

“…Checkpointing-recovery techniques make it possible for the workflow to resume execution from the last checkpoint instead of restarting from the beginning, should a failure occurs. Over-provisioning [11] techniques replicate a task on more than one resources to increase the probability of successful execution. Although these techniques address the reliability challenges to some extent, to the best of our knowledge, no large-scale study has been done on how effective they are when coupled with workflow management and scheduling.…”

Transforming the sensing and numerical prediction of high-impact local weather through dynamic adaptation

“…a forecast verification code) can be converted into and registered as services within LEAD and thus added to workflows. In order to ensure quality of service for time-critical meteorological experiments, especially forecasts (see §6), LEAD developed a fault tolerance recovery system (Fowler et al 2008;Kandaswamy et al 2008) based upon concepts such as over-provisioning and job resubmission.…”

Enhancing reliability with Latin Square redundancy on desktop grids.

“…[75] Most of the work in the area of scheduling has as its underlying paradigm the "task parallel" model of parallel computing. [55;76-81] In the Bag of Tasks (BoT) model the tasks are presumed to be independent and embarrassingly parallel.…”