Pegasus, a workflow management system for science automation

Deelman, Ewa; Vahi, Karan; Juve, Gideon; Rynge, Mats; Callaghan, S.; Maechling, P. J.; Mayani, Rajiv; Chen, Weiwei; Silva, Rafael Ferreira da; Livny, Miron; Wenger, Kent

doi:10.1016/j.future.2014.10.008

Cited by 680 publications

(443 citation statements)

References 59 publications

(72 reference statements)

Supporting

Mentioning

402

Contrasting

Unclassified

Order By: Relevance

“…• Pegasus [3] supports workflow executions in distributed environments, such as campus clusters, grids, clouds, and supercomputers. Pegasus maps an application onto available resources, optimizing the execution in terms of performance and data management.…”

Section: Classification Of Workflow Management Systemsmentioning

confidence: 99%

“…In recent years, numerous workflow management systems (WMSs) have been developed to manage the execution of diverse workflows on heterogeneous computing resources [3,4,5,6,7,8,9]. As user communities adopt and evolve WMSs to fit their own needs, many of the features and capabilities that were once common to most WMSs have become too distinct to share across systems.…”

Section: Introductionmentioning

confidence: 99%

“…For example, Taverna [8] and Galaxy [9] support advanced graphical user interfaces for workflow composition, making them suitable for bioinformatics researchers with little programming experience. Other systems, such as DAGMan [10] and Pegasus [3] offer scalability, robustness, and planning for heterogeneous high-throughput computation execution. For a new user, choosing the right WMS can be problematic simply because there are so many different WMSs and the selection criteria may not be obvious.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

Self Cite

119

View full text Add to dashboard Cite

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

Section: Classification Of Workflow Management Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

Self Cite

119

View full text Add to dashboard Cite

show abstract

“…For instance, in production Workflow Management Systems (WMSs) [1,2,3,4,5,6], the default behavior is that all output data is saved to files and all input data is read from files, which is exactly the CkptAll strategy. While this strategy leads to fast restarts in case of failures, its downside is that it maximizes checkpointing overhead.…”

Section: Introductionmentioning

confidence: 99%

Checkpointing Workflows for Fail-Stop Errors

Han¹,

Canon²,

Casanova³

et al. 2017

2017 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Abstract:We consider the problem of orchestrating the execution of workflow applications structured as Directed Acyclic Graphs (DAGs) on parallel computing platforms that are subject to fail-stop failures. The objective is to minimize expected overall execution time, or makespan. A solution to this problem consists of a schedule of the workflow tasks on the available processors and of a decision of which application data to checkpoint to stable storage, so as to mitigate the impact of processor failures. For general DAGs this problem is hopelessly intractable. In fact, given a solution, computing its expected makespan is still a difficult problem. To address this challenge, we consider a restricted class of graphs, Minimal Series-Parallel Graphs (M-SPGs). It turns out that many real-world workflow applications are naturally structured as M-SPGs. For this class of graphs, we propose a recursive list-scheduling algorithm that exploits the M-SPG structure to assign sub-graphs to individual processors, and uses dynamic programming to decide which tasks in these sub-gaphs should be checkpointed. Furthermore, it is possible to efficiently compute the expected makespan for the solution produced by this algorithm, using a first-order approximation of task weights and existing evaluation algorithms for 2-state probabilistic DAGs. We assess the performance of our algorithm for production workflow configurations, comparing it to (i) an approach in which all application data is checkpointed, which corresponds to the standard way in which most production workflows are executed today; and (ii) an approach in which no application data is checkpointed. Our results demonstrate that our algorithm strikes a good compromise between these two approaches, leading to lower checkpointing overhead than the former and to better resilience to failure than the latter. To the best of our knowledge, this is the first scheduling/checkpointing algorithm for workflow applications with fail-stop failures that considers workflow structures more general than mere linear chains of tasks.Key-words: workflow, checkpoint, fail-stop error, resilience. Stratégies de checkpoint pour les workflows en présence d'erreurs fatalesRésumé : Ce rapport considère l'ordonnancement de workflows (applications structurées en forme de graphes de tâches acycliques, ou DAGs) sur des plates-formes parallèlesà grandé echelle, soumisesà des erreurs fatales. L'objectif est de minimiser l'espérance du temps total d'exécution, ou makespan. Une solutionà ce problème comprend l'allocation ordonnée des tâches aux processeurs, et les décisions de checkpoint: quelles tâches sont suivies d'un checkpoint? Même pour une solution donnée, le calcul du makespan reste difficile. Nous nous restreignonsà une classe de DAGs particuliers, les graphes séries-parallèles minimaux, ou MSPGs. De nombreux workflows issus des applications ont pour graphe un M-SPG. Pour de tels graphes, nous proposons un algorithme qui utilise la structure récursive du M-SPG pour allouer des sous-graphesà chaque pro...

show abstract

“…When these simulations dataflows are managed by a parallel Scientific Workflow Management System (SWMS) [1], they benefit from provenance [2] and data parallelism among different programs that compose the workflow. Systems like Swift/T [3] and Pegasus [4] are highly scalable SWMS and have shown impressive performance results for many different scientific application domains [3,5].…”

mentioning

confidence: 99%

Raw data queries during data-intensive parallel workflow execution

Silva

Leite

Camata

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.

show abstract

Pegasus, a workflow management system for science automation

Cited by 680 publications

References 59 publications

A characterization of workflow management systems for extreme-scale applications

A characterization of workflow management systems for extreme-scale applications

Checkpointing Workflows for Fail-Stop Errors

Raw data queries during data-intensive parallel workflow execution

Contact Info

Product

Resources

About