A Framework for Collecting Provenance in Data-Centric Scientific Workflows

Simmhan, Yogesh; Plale, Beth; Gannon, Dennis

doi:10.1109/icws.2006.5

Cited by 92 publications

(73 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although several provenance models [7,11,2,4,10] have been proposed for scientific workflows, there has been no work on the provenance system that supports the notion of atomicity.…”

Section: Related Workmentioning

confidence: 99%

A Dataflow-Oriented Atomicity and Provenance System for Pipelined Scientific Workflows

Wang

Fei

et al. 2007

Computational Science – ICCS 2007

View full text Add to dashboard Cite

Abstract. Scientific workflows have gained great momentum in recent years due to their critical roles in e-Science and cyberinfrastructure applications. However, some tasks of a scientific workflow might fail during execution. A domain scientist might require a region of a scientific workflow to be "atomic". Data provenance, which determines the source data that are used to produce a data item, is also essential to scientific workflows. In this paper, we propose: (i) an architecture for scientific workflow management systems that supports both provenance and atomicity; (ii) a dataflow-oriented atomicity model that supports the notions of commit and abort; and (iii) a dataflow-oriented provenance model that, in addition to supporting existing provenance graphs and queries, also supports queries related to atomicity and failure.

show abstract

“…Although several provenance models [7,11,2,4,10] have been proposed for scientific workflows, there has been no work on the provenance system that supports the notion of atomicity.…”

Section: Related Workmentioning

confidence: 99%

A Dataflow-Oriented Atomicity and Provenance System for Pipelined Scientific Workflows

Wang

Fei

et al. 2007

Computational Science – ICCS 2007

View full text Add to dashboard Cite

show abstract

“…Recently, researchers have focused on workflow provenance [42,43]. Provenance data is an important source for better understanding the performance behavior of Grid workflows.…”

Section: Related Workmentioning

confidence: 99%

DIPAS: A distributed performance analysis service for grid service-based workflows

Truong

Brunner

Nae

et al. 2009

Future Generation Computer Systems

View full text Add to dashboard Cite

Grid workflows are executed on diverse resources whose interactions are highly complex and hardly predicted. Often the user and the workflow middleware services want to be informed about the performance behavior of workflows, as early as possible, so that they can steer the execution of workflows to compensate the performance loss or execution failures. This paper describes a distributed performance analysis service that supports tracing execution, analyzing performance overheads, and searching for performance problems of Web services-based workflows in the Grid. We present how the user and the Grid workflow middleware can utilize the distributed performance analysis service in order to optimize the execution of workflows.

show abstract

“…The Karma system [97,98] records provenance at four dimensions: execution, location, time, and dataflow, and uses a publishsubscribe notification protocol for provenance collection. Karma uses XML and relational database technologies to store and query provenance.…”

Section: Storing and Querying Scientific Workflow Provenancementioning

confidence: 99%

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Chebotko¹,

Lu²,

Fei³

et al. 2010

Data & Knowledge Engineering

View full text Add to dashboard Cite

Provenance metadata has become increasingly important to support scientific discovery reproducibility, result interpretation, and problem diagnosis in scientific workflow environments. The provenance management problem concerns the efficiency and effectiveness of the modeling, recording, representation, integration, storage, and querying of provenance metadata. Our approach to provenance management seamlessly integrates the interoperability, extensibility, and inference advantages of Semantic Web technologies with the storage and querying power of an RDBMS to meet the emerging requirements of scientific workflow provenance management. In this paper, we elaborate on the design of a relational RDF store, called RDFPROV, which is optimized for scientific workflow provenance querying and management. Specifically, we propose: i) two schema mapping algorithms to map an OWL provenance ontology to a relational database schema that is optimized for common provenance queries; ii) three efficient data mapping algorithms to map provenance RDF metadata to relational data according to the generated relational database schema, and iii) a schema-independent SPARQL-to-SQL translation algorithm that is optimized on-the-fly by using the type information of an instance available from the input provenance ontology and the statistics of the sizes of the tables in the database. Experimental results are presented to show that our algorithms are efficient and scalable. The comparison with two popular relational RDF stores, Jena and Sesame, and two commercial native RDF stores, AllegroGraph and BigOWLIM, showed that our optimizations result in improved performance and scalability for provenance metadata management. Finally, our case study for provenance management in a real-life biological simulation workflow showed the production quality and capability of the RDFPROV system. Although presented in the context of scientific workflow provenance management, many of our proposed techniques apply to general RDF data management as well.

show abstract

A Framework for Collecting Provenance in Data-Centric Scientific Workflows

Cited by 92 publications

References 25 publications

A Dataflow-Oriented Atomicity and Provenance System for Pipelined Scientific Workflows

A Dataflow-Oriented Atomicity and Provenance System for Pipelined Scientific Workflows

DIPAS: A distributed performance analysis service for grid service-based workflows

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Contact Info

Product

Resources

About