Pegasus: Mapping Scientific Workflows onto the Grid

Deelman, Ewa; Blythe, James M.; Gil, Yolanda; Kesselman, Carl; Mehta, Gaurang; Patil, Shivprasad; Su, Min; Vahi, Karan; Livny, Miron

doi:10.1007/978-3-540-28642-4_2

Cited by 393 publications

(273 citation statements)

References 13 publications

Supporting

Mentioning

260

Contrasting

Unclassified

Order By: Relevance

“…For example, user intervention may be required to provide tuning or convergence checks in order to adjust instrument settings and steer the analysis. In tomography workflows [66], 2D electron microscopic projection images are transformed to generate 3D volumes. During the feature tracking step in a particular workflow geared towards refining the correspondences [67], fine tuning can be performed using semiautomated methods that require user intervention.…”

Section: Semi-automated Workflowsmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

Self Cite

121

View full text Add to dashboard Cite

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

Section: Semi-automated Workflowsmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

Self Cite

121

View full text Add to dashboard Cite

show abstract

“…Los lenguajes de workflows permiten la composición de tales servicios. Con ellos, la aplicación de más alto nivel puede ser modelada en forma abstracta (Deelman et al, 2004;Hernández et al, 2004;Neubauer et al, 2006), donde los nodos representan tareas mientras que las aristas, relacionando tales nodos, representan dependencias entre tareas, flujo de datos o control de flujo. Asimismo, las tareas pueden ser realizadas por los servicios Grid básicos.…”

Section: Ict-unpa-26-2011unclassified

Separación avanzada de Concerns para desarrollar aplicaciones Grid

Casas¹,

Trejo²

2014

ICTUNPA

View full text Add to dashboard Cite

El desarrollo de aplicaciones para Grid computing, puede beneficiarse si se aplican enfoques emergentes de Ingeniería de Software basados en la Separación Avanzada de Concerns. En el presente trabajo analizamos los más representativos, de qué manera se aplicaron en el área de Grid computing. Los enfoques abordados incluyen Desarrollo de Software Dirigido por Mo-delos, Desarrollo de Software Basado en Componentes, Desarrollo de Software Orientado a Aspectos, Ingeniería de Línea de Productos y Desarrollo de Software Orientado a Features. Se analizan y comparan estos enfoques de Separación Avanzada de Concerns para determinar de qué manera y en qué grado cada uno de ellos satisfacen las necesidades que se requieren para desarrollo de aplicaciones para Grid computing.

show abstract

“…For example, Kepler provides mechanisms to create actors from web services, C/C++ applications, scripting languages, R 4 and Matlab, database queries, SRB 5 commands, and so on.…”

Section: The Kepler Scientific Workflow Systemmentioning

confidence: 99%

“…Scientific workflow systems [12,13,15,20,22,4] aim at facilitating these types of integration and analysis. 1 However, current scientific workflow systems still offer little or no support for effectively managing (and hiding) the inherent complexity of life-science data, leading to overly complex workflows that are hard to create, reuse, and optimize.…”

Section: Introductionmentioning

confidence: 99%

Collection-Oriented Scientific Workflows for Integrating and Analyzing Biological Data

McPhillips

Bowers

Ludäscher

2006

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Steps in scientific workflows often generate collections of results, causing the data flowing through workflows to become increasingly nested. Because conventional workflow components (or actors) typically operate on simple or application-specific data types, additional actors often are required to manage these nested data collections. As a result, conventional workflows become increasingly complex as data becomes more nested. This paper describes a new paradigm for developing scientific workflows that transparently manages nested data collections. Collection-oriented workflows have a number of advantages over conventional approaches including simpler workflow designs (e.g., requiring fewer actors and control-flow constructs) that are invariant under changes in data nesting. Our implementation within the Kepler scientific workflow system enables the explicit representation of collections and collection schemas, concurrent operation over collection contents via multi-level pipeline parallelism, and allows collection-aware actors to be composed readily from conventional actors.

show abstract

Pegasus: Mapping Scientific Workflows onto the Grid

Cited by 393 publications

References 13 publications

A characterization of workflow management systems for extreme-scale applications

A characterization of workflow management systems for extreme-scale applications

Separación avanzada de Concerns para desarrollar aplicaciones Grid

Collection-Oriented Scientific Workflows for Integrating and Analyzing Biological Data

Contact Info

Product

Resources

About