A data-driven workflow language for grids based on array programming principles

Montagnat, Johan; Isnard, Benjamin; Glatard, Tristan; Maheshwari, Ketan; Fornarino, Mireille Blay

doi:10.1145/1645164.1645171

Cited by 51 publications

(45 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SciFloware is data-driven [37] in the sense that its internal workflow language clearly separates the definition of data to be processed from the graph of activities to be applied to the data. This separation is particularly suited to scientific workflows where the same experiment has to be reused for analyzing different data sets without any change.…”

Section: Discussionmentioning

confidence: 99%

“…This separation is particularly suited to scientific workflows where the same experiment has to be reused for analyzing different data sets without any change. Optimization in SciFloware focuses on two main aspects: It uses asynchronous messages to execute workflows on a distributed infrastructure such as Grid or Cloud [37,38]. Furthermore, while it uses generally coarse grain parallelism, it is able to exploit the fact that some workflow actors may be algebraic operators (i.e., some actors are not black-boxes) to optimize the workflow execution.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

Plant phenotyping consists in the observation of physical and biochemical traits of plant genotypes in response to environmental conditions. Challenges, in particular in context of climate change and food security, are numerous. High-throughput platforms have been introduced to observe the dynamic growth of a large number of plants in different environmental conditions. Instead of considering a few genotypes at a time (as it is the case when phenomic traits are measured manually), such platforms make it possible to use completely new kinds of approaches. However, the data sets produced by such widely instrumented platforms are huge, constantly augmenting and produced by increasingly complex experiments, reaching a point where distributed computation is mandatory to extract knowledge from data.In this paper, we introduce InfraPhenoGrid, the infrastructure we designed and deploy to efficiently manage data sets produced by the PhenoArch plant phenomics platform in the context of the French Phenome Project. Our solution consists in deploying scientific workflows on a Grid using a middleware to pilot workflow executions. Our approach is user-friendly in the sense that despite the intrinsic complexity of the infrastructure, running scientific workflows and understanding results obtained (using provenance information) is kept as simple as possible for end-users.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…Many of them support counting loops without dependencies in grids and clusters, e.g. [1,7,18,21,30], some of which also support counting loops with dependencies [18,21,30]. Counting loops without dependencies can be represented as multiple DAG and counting loops with dependencies can also be linearized as a DAG [18].…”

Section: Related Workmentioning

confidence: 99%

“…Counting loops without dependencies can be represented as multiple DAG and counting loops with dependencies can also be linearized as a DAG [18]. For conditional loops, there are dataflow languages [21,30] that supports it out-of-the-box. The main issue of current iterative workflow approaches is that the workflow execution is tightly coupled to the workflow specification model.…”

Section: Related Workmentioning

confidence: 99%

“…However, when modeling an iterative application as a scientific workflow, the workflow language has to support iterative constructs. Current support for iteration in scientific workflows is designed for simpler loop behaviors [1,7,18,21,30]. They typically are control-flow-based and rely on three iteration types [10]: (1)counting loops without dependencies, (2) counting loops with dependencies and (3) conditional loops.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Data-centric iteration in dynamic workflows

Dias

Guerra

Rochinha

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

SummaryDynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the workflow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current support for iteration in scientific workflows is static and does not allow for changing data at runtime. In this paper, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to noniterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario.

show abstract

Chiron: a parallel engine for algebraic scientific workflows

Ogasawara

Dias

Silva

et al. 2013

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARY Large‐scale scientific experiments based on computer simulations are typically modeled as scientific workflows, which eases the chaining of different programs. These scientific workflows are defined, executed, and monitored by scientific workflow management systems (SWfMS). As these experiments manage large amounts of data, it becomes critical to execute them in high‐performance computing environments, such as clusters, grids, and clouds. However, few SWfMS provide parallel support. The ones that do so are usually labor‐intensive for workflow developers and have limited primitives to optimize workflow execution. To address these issues, we developed workflow algebra to specify and enable the optimization of parallel execution of scientific workflows. In this paper, we show how the workflow algebra is efficiently implemented in Chiron, an algebraic based parallel scientific workflow engine. Chiron has a unique native distributed provenance mechanism that enables runtime queries in a relational database. We developed two studies to evaluate the performance of our algebraic approach implemented in Chiron; the first study compares Chiron with different approaches, whereas the second one evaluates the scalability of Chiron. By analyzing the results, we conclude that Chiron is efficient in executing scientific workflows, with the benefits of declarative specification and runtime provenance support. Copyright © 2013 John Wiley & Sons, Ltd.

show abstract

A data-driven workflow language for grids based on array programming principles

Cited by 51 publications

References 19 publications

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Data-centric iteration in dynamic workflows

Chiron: a parallel engine for algebraic scientific workflows

Contact Info

Product

Resources

About