Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems

Deelman, Ewa; Singh, Gurmeet; Su, Min; Blythe, James M.; Gil, Yolanda; Kesselman, Carl; Mehta, Gaurang; Vahi, Karan; Berriman, G. Bruce; Good, J. C.; Laity, A. C.; Jacob, Joseph; Katz, Daniel S.

doi:10.1155/2005/128026

Cited by 946 publications

(756 citation statements)

References 30 publications

Supporting

Mentioning

747

Contrasting

Unclassified

Order By: Relevance

“…The average weight of a Montage task is 10s. Structurally, Montage is a three-level graph [28]. The first level (reprojection of input image) consists of a bipartite directed graph.…”

Section: Experimental Methodologymentioning

confidence: 99%

See 1 more Smart Citation

Checkpointing Workflows for Fail-Stop Errors

Han¹,

Canon²,

Casanova³

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Abstract:We consider the problem of orchestrating the execution of workflow applications structured as Directed Acyclic Graphs (DAGs) on parallel computing platforms that are subject to fail-stop failures. The objective is to minimize expected overall execution time, or makespan. A solution to this problem consists of a schedule of the workflow tasks on the available processors and of a decision of which application data to checkpoint to stable storage, so as to mitigate the impact of processor failures. For general DAGs this problem is hopelessly intractable. In fact, given a solution, computing its expected makespan is still a difficult problem. To address this challenge, we consider a restricted class of graphs, Minimal Series-Parallel Graphs (M-SPGs). It turns out that many real-world workflow applications are naturally structured as M-SPGs. For this class of graphs, we propose a recursive list-scheduling algorithm that exploits the M-SPG structure to assign sub-graphs to individual processors, and uses dynamic programming to decide which tasks in these sub-gaphs should be checkpointed. Furthermore, it is possible to efficiently compute the expected makespan for the solution produced by this algorithm, using a first-order approximation of task weights and existing evaluation algorithms for 2-state probabilistic DAGs. We assess the performance of our algorithm for production workflow configurations, comparing it to (i) an approach in which all application data is checkpointed, which corresponds to the standard way in which most production workflows are executed today; and (ii) an approach in which no application data is checkpointed. Our results demonstrate that our algorithm strikes a good compromise between these two approaches, leading to lower checkpointing overhead than the former and to better resilience to failure than the latter. To the best of our knowledge, this is the first scheduling/checkpointing algorithm for workflow applications with fail-stop failures that considers workflow structures more general than mere linear chains of tasks.Key-words: workflow, checkpoint, fail-stop error, resilience. Stratégies de checkpoint pour les workflows en présence d'erreurs fatalesRésumé : Ce rapport considère l'ordonnancement de workflows (applications structurées en forme de graphes de tâches acycliques, ou DAGs) sur des plates-formes parallèlesà grandé echelle, soumisesà des erreurs fatales. L'objectif est de minimiser l'espérance du temps total d'exécution, ou makespan. Une solutionà ce problème comprend l'allocation ordonnée des tâches aux processeurs, et les décisions de checkpoint: quelles tâches sont suivies d'un checkpoint? Même pour une solution donnée, le calcul du makespan reste difficile. Nous nous restreignonsà une classe de DAGs particuliers, les graphes séries-parallèles minimaux, ou MSPGs. De nombreux workflows issus des applications ont pour graphe un M-SPG. Pour de tels graphes, nous proposons un algorithme qui utilise la structure récursive du M-SPG pour allouer des sous-graphesà chaque pro...

show abstract

“…The average weight of a Montage task is 10s. Structurally, Montage is a three-level graph [28]. The first level (reprojection of input image) consists of a bipartite directed graph.…”

Section: Experimental Methodologymentioning

confidence: 99%

“…If instead n < p, then there is a surplus of processors. PropMap first assigns each input G i to one output M-SPG (Lines [27][28][29]. The p − n extra processors are then allocated iteratively to the output M-SPG with the largest weight (Lines 30-35).…”

Section: Scheduling M-spgsmentioning

confidence: 99%

Checkpointing Workflows for Fail-Stop Errors

Han¹,

Canon²,

Casanova³

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

show abstract

“…To evaluate the approach to adaptive workflow execution using utility functions, we have extended the Pegasus workflow management system [3] with a framework based around the MAPE functional decomposition [9] which partitions adaptive functionality into four components, Monitoring, Analysis, Planning and Execution. The principal components of relevance to the experiments, and their relationships, are illustrated in Figure 3.…”

Section: A Experimental Contextmentioning

confidence: 99%

“…Such static decision making involves the risk that decisions may be made on the basis of information about resource performance and availability that quickly becomes outdated. As a result, benefits may result either from incremental compilation, whereby resource allocation decisions are made for part of a workflow at a time (e.g., [3]), or by dynamically revising compilation decisions that gave rise to a concrete workflow while it is executing (e.g., [4], [5], [6], [7]). In principle, any decision that was made statically during workflow compilation can be revisited at runtime [8].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Supporting IPv6 Interaction with Wireless Sensor Networks Using NP++

Jakeman

Hughes

Coulson

et al. 2008

Wireless Algorithms, Systems, and Applications

View full text Add to dashboard Cite

Abstract-Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has responsibility for establishing how best to map tasks within a workflow to the available resources. As workflows are typically run over shared resources, and thus face unpredictable and changing resource capabilties, there may be benefit to be derived from adapting the task-to-resource mapping while a workflow is executing. This paper describes the use of utility functions to express the relative merits of alternative mappings; in essence, a utility function can be used to give a score to a candidate mapping, and the exploration of alternative mappings can be cast as an optimization problem. In this approach, changing the utility function allows adaptations to be carried out with a view to meeting different objectives. The contributions of this paper include: (i) a description of how adaptive workflow execution can be expressed as an optimization problem where the objective of the adaptation is to maximize some property expressed as a utility function; (ii) a description of how the approach has been applied to support adaptive workflow execution in grids; and (iii) an experimental evaluation of the resulting approach for alternative utility measures based on response time and profit.

show abstract

Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance

Gil

David

Demir

et al. 2016

Earth and Space Science

Self Cite

155

168

View full text Add to dashboard Cite

Geoscientists now live in a world rich with digital data and methods, and their computational research cannot be fully captured in traditional publications. The Geoscience Paper of the Future (GPF) presents an approach to fully document, share, and cite all their research products including data, software, and computational provenance. This article proposes best practices for GPF authors to make data, software, and methods openly accessible, citable, and well documented. The publication of digital objects empowers scientists to manage their research products as valuable scientific assets in an open and transparent way that enables broader access by other scientists, students, decision makers, and the public. Improving documentation and dissemination of research will accelerate the pace of scientific discovery by improving the ability of others to build upon published work.

show abstract

Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems

Cited by 946 publications

References 30 publications

Checkpointing Workflows for Fail-Stop Errors

Checkpointing Workflows for Fail-Stop Errors

Supporting IPv6 Interaction with Wireless Sensor Networks Using NP++

Toward the Geoscience Paper of the Future: Best practices for documenting and sharing research from data to software to provenance

Contact Info

Product

Resources

About