Tracing and using data lineage for pipeline processing in Astro-WISE

Abstract

Most workflow systems that support data provenance primarily focus on tracing lineage of data. Data provenance by data lineage provides the derivation history of data including information about services and input data that contributed to the creation of a data product. We show that tracing lineage by means of full backward chaining not only enables users to share, discover and reuse the data, but also supports scientific data processing through storage, retrieval and (re)processing of digitized scientific data. In this paper, we present Astro-WISE, a distributed system for processing, analyzing and disseminating wide field imaging astronomical data. We show how Astro-WISE traces lineage of data and how it facilitates data processing, retrieval, storage and archiving. Particularly we show how it solves issues related to the changing data items typical for the scientific environment, such as physical changes in calibrations, our insight in these changes and improved methods for deriving results.

Sub-image data processing in Astro-WISE

“…Astro-WISE enables astronomers to perform scientific experiments in a distributed environment. The full description of lineage tracing in Astro-WISE is described in [21]. Astro-WISE tracks all parameters used (or created) during runtime that induces any dependency relationship between objects, i.e., parameters that would affect the state of an object.…”

“…Let us consider the RawFrame and a list of sources as the only data products that are not part of lineage data, then any other data product (e.g., metadata, image files, other data objects) will be lineage data. From [21], data lineage is stored as persistent objects in a database. Lineage data is stored as fully integrated objects or as descriptors.…”

“…Since we are working with pixels, this framework required lineage at pixel level. We extended our lineage model presented in [21] to trace lineage at pixel level and then use pixel lineage for sub-image processing.…”

“…The goal of this paper is twofold; firstly, we extend our lineage model presented in [21] to trace lineage at pixel level and we show that it accurately locates all pixels that are involved in the processing of a result. Secondly, we show that using lineage we are able to support sub-image processing.…”