Linking multiple workflow provenance traces for interoperable collaborative science

Abstract

Abstract-Scientific collaboration increasingly involves data sharing between separate groups. We consider a scenario where data products of scientific workflows are published and then used by other researchers as inputs to their workflows. For proper interpretation, shared data must be complemented by descriptive metadata. We focus on provenance traces, a prime example of such metadata which describes the genesis and processing history of data products in terms of the computational workflow steps. Through the reuse of published data, virtual, implicitly collaborative experiments emerge, making it desirable to compose the independently generated traces into global ones that describe the combined executions as single, seamless experiments. We present a model for provenance sharing that realizes this holistic view by overcoming the various interoperability problems that emerge from the heterogeneity of workflow systems, data formats, and provenance models. At the heart lie (i) an abstract workflow and provenance model in which (ii) data sharing becomes itself part of the combined workflow. We then describe an implementation of our model that we developed in the context of the Data Observation Network for Earth (DataONE) project and that can "stitch together" traces from different Kepler and Taverna workflow runs. It provides a prototypical framework for seamless cross-system, collaborative provenance management and can be easily extended to include other systems. Our approach also opens the door to new ways of workflow interoperability not only through often elusive workflow standards but through shared provenance information from public repositories.

A declarative approach to customize workflow provenance

“…Scientific workflow systems often automatically record provenance data [15,6], and a provenance graph may resemble the workflow graph, i.e., the former can be seen as an instance of the latter [17]. Using the framework discussed above, we customize provenance graphs to remove private and irrelevant information.…”

Reconciling Provenance Policy Conflicts by Inventing Anonymous Nodes

“…Scientific workflow systems often automatically record such provenance [12,13], and the provenance graphs may resemble the workflow graph, i.e., the former can be seen as instances of the latter [2]. At the workflow specification level, actors are used to represent the computational steps, implemented by software components, while at the provenance (or instance) level, we have invocations of those actors.…”

“…In the emerging paradigm of collaborative, data-intensive science, sharing data products even prior to publication may be desirable [1,2]. Yet, without a proper scientific publication associated with shared data, its validity and accuracy is difficult to assess.…”

Search, adapt, and reuse

“…1. Currently, essentially every SciWFM has their particular language for specifying a workflow, equipped with a particular semantics of how such a specification is interpreted as an executable program [MLB+10]. Things get even more complicated with systems that allow dynamic adaptation of the semantics of a specification; for instance, the Kepler system offers different so-called directors for orchestrating workflow execution, leading to entirely different execution threads [DKM+05].…”