Examining the Challenges of Scientific Workflows

Gil, Yolanda; Deelman, Ewa; Ellisman, Mark H.; Fahringer, Thomas; Fox, Geoffrey; Gannon, Dennis; Goble, Carole; Livny, Miron; Moreau, Luc; Myers, James D.

doi:10.1109/mc.2007.421

Cited by 436 publications

(308 citation statements)

References 3 publications

Supporting

Mentioning

303

Contrasting

Unclassified

Order By: Relevance

“…They have recently gained attention as a flexible programming paradigm for modeling, representing, and executing complex computations and analysis pipelines in many different areas of scientific research [17]. A considerable number of scientific workflow management systems (SWfMS) has been developed, including Taverna [18], Kepler [19], and Pegasus [20].…”

Section: Introductionmentioning

confidence: 99%

DynamicCloudSim: Simulating heterogeneity in computational clouds

Bux

Leser

2015

Future Generation Computer Systems

View full text Add to dashboard Cite

Simulation has become a commonly employed first step in evaluating novel approaches towards resource allocation and task scheduling on distributed architectures. However, existing simulators fall short in their modeling of the instability common to shared computational infrastructure, such as public clouds. In this work, we present DynamicCloudSim which extends the popular simulation toolkit CloudSim with several factors of instability, including inhomogeneity and dynamic changes of performance at runtime as well as failures during task execution. As a use case and validation of the introduced functionality, we simulate the impact of instability on scientific workflow scheduling by assessing and comparing the performance of four schedulers in the course of several experiments. Results indicate that our model seems to adequately capture the most important aspects of cloud performance instability, though a validation on real hardware is still pending. The source code of DynamicCloudSim and the examined schedulers is available at https://code.google.com/p/dynamiccloudsim/.

show abstract

Section: Introductionmentioning

confidence: 99%

DynamicCloudSim: Simulating heterogeneity in computational clouds

Bux

Leser

2015

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…To tackle the exploratory nature of science and the dynamic process involved in scientific analysis, dynamic workflows have been identified as an open challenge [13]. Dynamic workflows are scientific workflows that are subject to rapid reuse and exploration accompanied by continuous adaptation and improvement [13].…”

Section: Introductionmentioning

confidence: 99%

Data-centric iteration in dynamic workflows

Dias

Guerra

Rochinha

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

SummaryDynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the workflow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current support for iteration in scientific workflows is static and does not allow for changing data at runtime. In this paper, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to noniterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario.

show abstract

“…This runtime data analysis is the basis for workflow steering [15]. Our preliminary experiments in [11] have shown the potential of user steering features in workflows for tracking, evaluating and adapting the execution of a workflow.…”

Section: Introductionmentioning

confidence: 99%

“…This steering mechanism allows for optimizing workflow execution and has gained importance in the last years since HPC environments, such as clouds, are paid according to the time used. In fact, this is highlighted by Gil et al [15] as "a fundamental technology to fully support e-Science".…”

Section: Introductionmentioning

confidence: 99%

Using Domain-Specific Data to Enhance Scientific Workflow Steering Queries

Gonçalves

Oliveira

Ocaña

et al. 2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In scientific workflows, provenance data helps scientists in understanding, evaluating and reproducing their results. Provenance data generated at runtime can also support workflow steering mechanisms. Steering facilities for workflows is considered a challenge due to its dynamic demands during execution. To steer, for example, scientists should be able to suspend (or stop) a workflow execution when the approximate solution meets (or deviates) preset criteria. These criteria are commonly evaluated based on provenance data (execution data) and domain-specific data. We claim that the final decision on whether to interfere on the workflow execution may only become feasible when workflows can be steered by scientists using provenance data enriched with domain-specific data. In this paper we propose an approach based on specialized software components, named Data Extractor (DE), to acquire domain-specific data from data files produced during a scientific workflow execution. DE gathers domain-specific data from produced data files and associates it to existing provenance data on the provenance repository. We have evaluated the proposed approach using a real bioinformatics workflow for comparative genomics executed in SciCumulus cloud workflow parallel engine.

show abstract

Examining the Challenges of Scientific Workflows

Abstract: Abstract

Cited by 436 publications

References 3 publications

DynamicCloudSim: Simulating heterogeneity in computational clouds

DynamicCloudSim: Simulating heterogeneity in computational clouds

Data-centric iteration in dynamic workflows

Using Domain-Specific Data to Enhance Scientific Workflow Steering Queries

Contact Info

Product

Resources

About