Accelerating the scientific exploration process with scientific workflows

Abstract

Although an increasing amount of middleware has emerged in the last few years to achieve remote data access, distributed job execution, and data management, orchestrating these technologies with minimal overhead still remains a difficult task for scientists. Scientific workflow systems improve this situation by creating interfaces to a variety of technologies and automating the execution and monitoring of the workflows. Workflow systems provide domain-independent customizable interfaces and tools that combine different tools and technologies along with efficient methods for using them. As simulations and experiments move into the petascale regime, the orchestration of long running data and compute intensive tasks is becoming a major requirement for the successful steering and completion of scientific investigations. A scientific workflow is the process of combining data and processes into a configurable, structured set of steps that implement semi-automated computational solutions of a scientific problem. Kepler is a cross-project collaboration, co-founded by the SciDAC Scientific Data Management (SDM) Center, whose purpose is to develop a domain-independent scientific workflow system. It provides a workflow environment in which scientists design and execute scientific workflows by specifying the desired sequence of computational actions and the appropriate data flow, including required data transformations, between these steps. Currently deployed workflows range from local analytical pipelines to distributed, high-performance and high-throughput applications, which can be both data-and compute-intensive. The scientific workflow approach offers a number of advantages over traditional scripting-based approaches, including ease of configuration, improved reusability and maintenance of workflows and components (called actors), automated provenance management, "smart" re-running of different versions of workflow instances, on-the-fly updateable parameters, monitoring of long running tasks, and support for fault-tolerance and recovery from failures. We present an overview of common scientific workflow requirements and their associated features which are lacking in current state-of-the-art workflow management systems. We then illustrate features of the Kepler workflow system, both from a user's and a "workflow engineer's" point-of-view. In particular, we highlight the use of some of the current features of Kepler in several scientific applications, as well as upcoming extensions and improvements that are geared specifically for SciDAC user communities.

Scientific Workflow Interchanging through Patterns: Reversals and Lessons Learned

“…Most of these remaining tasks would not be needed in a (semantically) equivalent workflow specification in the Kepler system [5], for example.…”

“…3 Some of these patterns have their specifications in our language enriched with the data-oriented dependency relationships identified in [15]: Sequence: binds a single output port to a single input port; Parallel Split: binds a single output port to two or more input ports, replicating the same data from the output port to all input ports; Simple Merge: binds two or more output ports to a single input port, feeding the input port with data received from each output port in an interleaved way; Synchronization: similar in structure to the Simple Merge pattern, but the task with the input port may be only executed when data coming from all the output ports have been received and grouped according to some criteria; Exclusive Choice: similar in structure to the Parallel Split pattern, but only one of the input ports may receive data from the output port, according to some condition. Besides dependency relationships, we also needed a way to describe component types in the style specification so as to represent workflow tasks.…”

“…bioinformatics [1], astronomy [2] and physics [3]. In a nutshell, a workflow may be described as a directed graph with its vertex set representing tasks and its edge set representing dependency relationships between the tasks.…”

“…http://crowdlabs.org/vistrails/ which is specific to VisTrails workflows-but at the time of this writing we were unable to explore these other repositories 5. A web-based interface for our tool and also for giving access to our language specification is available at https://www.lncc.br/sinapad/wisp-portal/…”

Intelligent grid enabled services for neuroimaging analysis

“…Through this service users can specify their pipelines (or workflows, connected collections of algorithms). This service is designed to support all major neuroimaging workflow authoring environments (such as LONI Pipeline [34], Kepler [35] etc.) and with the help of the Glueing Service it can enable their enactment on a number of underlying infrastructures.…”