Four level provenance support to achieve portable reproducibility of scientific workflows

Abstract

-In the scientist's community one of the most vital challenges is the issue of reproducibility of workflow execution. In order to reproduce the results of an experiment, on one hand provenance information must be collected and on the other hand the dependencies of the execution need to be eliminated. Concerning the workflow execution environment we have differentiated four levels of provenance: infrastructural, environmental, workflow and data provenance. During the re-execution at all levels the components can change and capturing the data of each levels targets different problems to solve. For example storing the environmental and infrastructural parameters enables the portability of workflows between the different parallel and distributed systems (grid, HPC, cloud). The describers of the workflow model enable tracking the different versions of the workflow and their impacts on the execution. Our goal is to capture the most optimal parameters in number and type as well and reconstruct the way of data production independently from the environment. In this paper we investigate the necessary and satisfactory parameters of workflow reproducibility and give a mathematical formula to determine the rate of reproducibility. These measurements allow the scientist to make a decision about the next steps toward the creation of reproducible workflows.

Designing for Reproducibility

“…Those include documentation of all relevant analysis artefacts. In their paper, Bánáti et al [3] classified several dependencies -that have a direct impact on the reproducibility of experiments -into three categories: infrastructural dependency, data dependency and job execution dependency. According to their work, reproducibility of computational studies requires to fully document the computational environments, and to ensure that all experimental resources remain accessible.…”

Minimal sufficient information about the scientific workflows to create reproducible experiment

“…With help of the given results together with the information gained from the user the system can create a so called Dependency Dataset, which will store all the jobs which depend on any external circumstances and may not be reproducible. In our previous paper [2] we showed, that the rate of reproducibility of a scientific workflow can be computed with the help of which the reproducible parts of workflow can be determined. From this dataset, after viewing the results the user -before finally submits his workflow -can think over the model, he can modify it and can eliminate certain dependencies or he can decide to apply extra provenance or virtualization tools to preserve the workflow.…”

“…In our previous work [2] we determined the four level of the provenance, and the different utilizations of the captured data in the different levels. Capturing provenance data during the running time of the workflow is crucial to create reproducible workflows.…”

“…A workflow execution may depend on volatile third party resources and services; special hardware or software elements which are available only in a few and special infrastructure; deadlines, which cannot be accomplished on every infrastructure or it can be based on non-deterministic computation which apply for example random generated values [2].…”

Classification of scientific workflows based on reproducibility analysis

“…The former can be perceived as the necessary and the latter one as the satisfactory requirements of the reproducibility. The dependencies of the execution mean those resources which require external (out of the scientific workflow management system, SWfMS) services or resources such as third party services, special hardwares/softwares or random value generator [2]. Elimination of these dependencies in most cases is not possible, so they have to be handled in some other way: different methods should be set up to make the workflows reproducible.…”