A survey of data provenance in e-science

Abstract

Data management is growing in complexity as largescale applications take advantage of the loosely coupled resources brought together by grid middleware and by abundant storage capacity. Metadata describing the data products used in and generated by these applications is essential to disambiguate the data and enable reuse. Data provenance, one kind of metadata, pertains to the derivation history of a data product starting from its original sources. In this paper we create a taxonomy of data provenance characteristics and apply it to current research efforts in e-science, focusing primarily on scientific workflow approaches. The main aspect of our taxonomy categorizes provenance systems based on why they record provenance, what they describe, how they represent and store provenance, and ways to disseminate it. The survey culminates with an identification of open research problems in the field.

Building Scientific Workflow with Taverna and BPEL: A Comparative Study in caGrid

“…The final phase of the workflow lifecycle, namely analysis of the results, is increasingly perceived as of great importance within the e-science community [15]. The provenance of a piece of data produced by an arbitrary process is a complete account of how that piece of data was computed, starting from user input and taking into account intermediate results produced by the processors involved in the computation.…”

Looking Inside the Black-Box: Capturing Data Provenance Using Dynamic Instrumentation

“…For a comprehensive overview of the field, we refer the reader to Moreau [24]. Furthermore, Cheney et al [5] and Simmhan et al [30] provide specialized reviews for databases and e-science respectively. Here, we focus on systems for provenance capture.…”

Provenance management in curated databases

“…One form of provenance is "workflow" or "coarse-grained" provenance: information describing how derived data has been calculated from raw observations [3,10,14,21]. Workflow provenance is important in scientific computation, but is not a major concern in curated databases.…”

“…As mentioned in the introduction, "workflow" or "coarse-grained" provenance has been studied extensively in the context of scientific computation [10,14,26]; Bose and Frew [3] and Simmhan et al [21] survey most existing research on such systems. These approaches record the process used to derive processed data products from raw data.…”