Experience in using a process language to define scientific workflow and generate dataset provenance

Abstract

This paper describes our experiences in exploring the applicability of software engineering approaches to scientific data management problems. Specifically, this paper describes how process definition languages can be used to expedite production of scientific datasets as well as to generate documentation of their provenance. Our approach uses a process definition language that incorporates powerful semantics to encode scientific processes in the form of a Process Definition Graph (PDG). The paper describes how execution of the PDG-defined process can generate Dataset Derivation Graphs (DDGs), metadata that document how the scientific process developed each of its product datasets. The paper uses an example to show that scientific processes may be complex and to illustrate why some of the more powerful semantic features of the process definition language are useful in supporting clarity and conciseness in representing such processes. This work is similar in goals to work generally referred to as Scientific Workflow. The paper demonstrates the contribution that software engineering can make to this domain.

Smart checklists for human-intensive medical systems

“…Initially SmartCheck will use Data Derivation Graphs (DDGs) [13], [14] to manage such process information. A DDG is a DAG-like structure that records the steps performed, what entity performed them, and (usually pointers to) the artifacts used as inputs and outputs.…”

DEX: Digital evidence provenance supporting reproducibility and comparison

“…These graphs are defined and queried using a Virtual Data Language. Osterweil et al [17] similarly apply graphical notions of provenance to hydrological studies using derivation functions that include both tools and human operations. In digital forensics, provenance techniques can similarly be applied to report on transformations from an acquired raw image (i.e., a data source) to evidence (i.e., a data product); Process provenance metadata is not a standard part of investigations though it would address many concerns about the reproducibility of examinations.…”

A Pattern for Modeling Rework in Software Development Processes

“…In this section we present several concrete examples of rework that we have encountered, and which seem amenable to solution using the pattern presented later in this paper. Though we have found rework in dispute resolution [2], scientific data processing [3,4], health care delivery [5], and other domains, the examples here are drawn from software engineering.…”

“…Since rework often involves the modification of existing entities, is is usual to pass these entities by reference. We note, however, that the derivation history [9,3,4] of these entities is often particularly important in the creation of a rework plan as that history is often necessary in order to guide the re-invocation away from repetition of choices that have previously been shown to have led to later problems.…”

“…Check Failed that includes the failed element. The trigger is the input to Create Define Requirement Context which creates a context including the element to be passed 4 to the re-instantiation of Define Requirement Element 5 and finally, control is returned to Check Requirement Element 6 .…”