Scientific workflow management and the Kepler system

Abstract

Many scientific disciplines are now data and information driven, and new scientific knowledge is often gained by scientists putting together data analysis and knowledge discovery "pipelines". A related trend is that more and more scientific communities realize the benefits of sharing their data and computational services, and are thus contributing to a distributed data and computational community infrastructure (a.k.a. "the Grid"). However, this infrastructure is only a means to an end and scientists ideally should be bothered little with its existence. The goal is for scientists to focus on development and use of what we call scientific workflows. These are networks of analytical steps that may involve, e.g., database access and querying steps, data analysis and mining steps, and many other steps including computationally intensive jobs on high performance cluster computers. In this paper we describe characteristics of and requirements for scientific workflows as identified in a number of our application projects. We then elaborate on Kepler, a particular scientific workflow system, currently under development across a number of scientific data management projects. We describe some key features of Kepler and its underlying Ptolemy ii system, planned extensions, and areas of future research. Kepler is a communitydriven, open source project, and we always welcome related projects and new contributors to join. *

A Formal Framework for Provenance Security

“…Kepler [8] and Taverna [7] among many others); the Open Provenance Model [14] is a popular format for this kind of provenance but in this paper, we will focus on a simple form of OPM-like graphs. Our presentation draws primarily on the model introduced by Davidson et al [?]…”

“…For workflow provenance, many implementations exist [7], [8] but formal definitions of the meaning of workflow programs have only started to appear recently (see for example [33], [34]), while the provenance semantics of these tools is usually specified informally, at best [7]. As a result there is a variety of models and styles of provenance for workflows.…”

“…Among these, the first three seem to address subjective concerns where there is not necessarily a unique way of framing the problem or deriving solutions. The many different models for and implementations of provenance in different systems, such as databases [12], [13], workflow management systems [7], [8], and storage systems [10], [11], suggests that there may be no single definition that applies to all these situations, nor an obvious choice of "best" or most general provenance mechanism even for a specific system. Indeed, it seems likely that the answer to question (2) is actually "no", since any real system can be modeled at finer and finer levels of granularity.…”

Automated Component–Level Evaluation: Present and Future

“…The more components that can be called the better the acceptance of such a system will be. A possible implementation of such complex IR systems could be through a workflow paradigm, following the lead of eScience with systems such as Kepler 6 [17]. It might be beneficial to have a centrally managed infrastructure where components can be made available also from groups that lack the computing power to host components.…”

Collection-Oriented Scientific Workflows for Integrating and Analyzing Biological Data

“…Scientific workflow systems [12,13,15,20,22,4] aim at facilitating these types of integration and analysis. 1 However, current scientific workflow systems still offer little or no support for effectively managing (and hiding) the inherent complexity of life-science data, leading to overly complex workflows that are hard to create, reuse, and optimize.…”