Data Provenance

Abstract

The provenance of research data is of critical importance to the reproducibility of and trust in scientific results. As research infrastructures provide more amalgamated datasets for researchers and more integrated facilities for processing and publishing data, the capture of provenance in a standard, machine-actionable form becomes especially important. Significant progress has already been made in providing standards and tools for provenance tracking, but the integration of these technologies into research infrastructure remains limited in many scientific domains. Further development and collaboration are required to provide frameworks for provenance capture that can be adopted by as widely as possible, facilitating interoperability as well as dataset reuse. In this chapter, we examine the current state of the art for provenance, and the current state of provenance capture in environmental and Earth science research infrastructures in Europe, as surveyed in the course of the ENVRIplus project. We describe a service developed for the upload, dissemination and application of provenance templates that can be used to generate standardised provenance traces from input data in accordance with current best practice and standards. The use of such a service by research infrastructure architects and researchers can expedite both the understanding and use of provenance technologies, and so drive the standard use of provenance capture technologies in future research infrastructure developments.

Automatic Generation of Temporal Data Provenance From Biodiversity Information Systems

“…Similar to workflow-based solutions, template-based solutions, which rely on the definition of provenance templates, are highly domain-specific. Therefore, such solutions perfectly fit in well-defined domains such as health (Curcin et al, 2017), software engineering (Sáenz-Adán et al, 2018), and environmental and earth sciences (Magagna et al, 2020).…”

“…In software engineering, a template-based provenance capture mechanism can also be implemented by mapping the structural diagrams of designed applications into provenance templates (Sáenz-Adán et al, 2018). Furthermore, a service for uploading and disseminating provenance templates has been established in the field of environmental and earth sciences, which can be used to build uniform provenance traces from input data in accordance with standards (Magagna et al, 2020).…”

Provenance-enhanced Root Cause Analysis for Jupyter Notebooks

“…Provenance can be used in scientific computing to track the evolution of different research assets, e.g., metadata, systems, workflow, or data [22]. Provenance metadata describes an arbitrary process modeled by an arbitrary model with metadata [23].…”

Distributed in-memory data management for workflow executions

“…Scientific workflows need provenance data, since they allow for analysis, quality assessment, reliability, reproducibility, and reusability of the scientific results ( Davidson & Freire, 2008 ; Freire et al, 2008 ). Provenance data track which and how processes were utilized to derive each data entity; keep data authorship; and provide powerful means for analytical queries to perform data reduction, discovery and interpretation ( Magagna et al, 2020 ). Such features are considered as important as the scientific achievement itself, since experiment’s reliability can be compromised otherwise ( Freire et al, 2008 ).…”