Provenance Collection Support in the Kepler Scientific Workflow System

Altıntaş, Ilkay; Barney, Oscar; Jaeger-Frank, Efrat

doi:10.1007/11890850_14

Cited by 221 publications

(188 citation statements)

References 17 publications

Supporting

Mentioning

185

Contrasting

Unclassified

Order By: Relevance

“…Altintas, et al's extension of the Kepler Scientific Workflow System [9] is also similar to the work presented here. Just as we may extend existing propagator networks to support provenance, Altintas demonstrates an extension of an existing framework, Kepler, to support provenance.…”

Section: Related Worksupporting

confidence: 74%

Data Provenance in Distributed Propagator Networks

Jacobi

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The heterogeneous and unreliable nature of distributed systems has created a distinct need for the inclusion of provenance within their design to allow for error correction and redundancy. Many traditional distributed systems have limited provenance tracing abilities, usually included in generic workflow generation or in an application-specific way. The novel programming paradigm of distributed propagator networks allows for the inclusion of provenance from the ground up. In this paper, I present the concept of propagator networks and demonstrate how provenance may be easily integrated into programs built using them. I also demonstrate the possibility of converting non-provenanceaware applications built using propagator networks into provenanceaware applications by simply performing a transformation of the existing program structure.

show abstract

Section: Related Worksupporting

confidence: 74%

Data Provenance in Distributed Propagator Networks

Jacobi

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Each flavour of dataflow computation is manifested in Kepler as a Director, that dictates the mechanics of dataflow among analytical steps. Kepler utilises provenance to perform smart workflow re-runs [20], where subparts of a workflow can be selectively re-executed using cached outputs of upstream portions of a workflow.…”

Section: Factorial Design In Workflow Systemsmentioning

confidence: 99%

Static analysis of Taverna workflows to predict provenance patterns

Alper

Belhajjame

Goble

2017

Future Generation Computer Systems

View full text Add to dashboard Cite

Workflows have found adoption in scientific domains particularly due to their automation and provenance features. Using workflows scientists can repeat analyses with different input parameters to later use provenance to access and compare results based on respective parameters. An assumption that is often made is that by designing an analysis as a workflow, we get parameterto-result traceability for free with workflow provenance. This assumption holds for cases of coarse-grained traceability, where an entire workflow is subjected to repetition and all workflow parameters contribute to all results. On the other hand for cases requiring finer grained traceability, where a workflow is configured with collections of parameters and analyses within a workflow are repeated with combinations of parameters from collections, this assumption is not guaranteed to hold. In this paper we identify two dimensions that affect fine-grained traceability as: 1) the level of granularity supported by a workflow system in modelling parameters/data in workflows and in provenance, which we name as the level of support for Factorial Design, and 2) the practice of scientists in successfully encoding Factorial Design into workflows. Taverna is a workflow system that provides extensive features for factorial design, meanwhile it provides an uncontrolled approach to workflow design; meaning scientists may create workflows, which, when run, could break traceability in provenance. Using a real-world Taverna workflow we show how broken traceability manifests in provenance and how it can render provenance practically useless for accessing workflow outputs derived from particular input parameters. In order to prevent broken traceability from occurring, we describe a rule-based static analysis technique, which operates over workflow descriptions and anticipates patterns in provenance. Our rules exploit the well-defined execution behaviour in the Taverna system. In order to understand Factorial Design support in workflow systems in general, we provide a comparative survey. We conclude that other workflow systems also provide constructs for Factorial Design, and, similar to Taverna, they too are prone to broken traceability.

show abstract

“…The Kepler system [10,70] implements a provenance recorder to record information about a workflow run, including the context, data derivation history, workflow definition, and workflow evolution. The provenance recorder is parametric and customizable, allowing the user to choose different levels of granularity of provenance metadata for recording.…”

Section: Storing and Querying Scientific Workflow Provenancementioning

confidence: 99%

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Chebotko¹,

Lu²,

Fei³

et al. 2010

Data & Knowledge Engineering

View full text Add to dashboard Cite

Provenance metadata has become increasingly important to support scientific discovery reproducibility, result interpretation, and problem diagnosis in scientific workflow environments. The provenance management problem concerns the efficiency and effectiveness of the modeling, recording, representation, integration, storage, and querying of provenance metadata. Our approach to provenance management seamlessly integrates the interoperability, extensibility, and inference advantages of Semantic Web technologies with the storage and querying power of an RDBMS to meet the emerging requirements of scientific workflow provenance management. In this paper, we elaborate on the design of a relational RDF store, called RDFPROV, which is optimized for scientific workflow provenance querying and management. Specifically, we propose: i) two schema mapping algorithms to map an OWL provenance ontology to a relational database schema that is optimized for common provenance queries; ii) three efficient data mapping algorithms to map provenance RDF metadata to relational data according to the generated relational database schema, and iii) a schema-independent SPARQL-to-SQL translation algorithm that is optimized on-the-fly by using the type information of an instance available from the input provenance ontology and the statistics of the sizes of the tables in the database. Experimental results are presented to show that our algorithms are efficient and scalable. The comparison with two popular relational RDF stores, Jena and Sesame, and two commercial native RDF stores, AllegroGraph and BigOWLIM, showed that our optimizations result in improved performance and scalability for provenance metadata management. Finally, our case study for provenance management in a real-life biological simulation workflow showed the production quality and capability of the RDFPROV system. Although presented in the context of scientific workflow provenance management, many of our proposed techniques apply to general RDF data management as well.

show abstract

Provenance Collection Support in the Kepler Scientific Workflow System

Cited by 221 publications

References 17 publications

Data Provenance in Distributed Propagator Networks

Data Provenance in Distributed Propagator Networks

Static analysis of Taverna workflows to predict provenance patterns

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Contact Info

Product

Resources

About