SubZero: A fine-grained lineage system for scientific databases

Abstract

Abstract-Data lineage is a key component of provenance that helps scientists track and query relationships between input and output data. While current systems readily support lineage relationships at the file or data array level, finer-grained support at an array-cell level is impractical due to the lack of support for user defined operators and the high runtime and storage overhead to store such lineage.We interviewed scientists in several domains to identify a set of common semantics that can be leveraged to efficiently store fine-grained lineage. We use the insights to define lineage representations that efficiently capture common locality properties in the lineage data, and a set of APIs so operator developers can easily export lineage information from user defined operators. Finally, we introduce two benchmarks derived from astronomy and genomics, and show that our techniques can reduce lineage query costs by up to 10× while incuring substantially less impact on workflow runtime and storage.

LogGC

“…There exists a line of research in providing fine-grain data lineage for database systems. Trio [2] and SubZero [25] introduce new features to manage fine-grain lineage along with data. They track provenance by transforming/reversing queries.…”

Provenance Research Issues and Challenges in the Big Data Era

“…Data provenance concerns with the problem of detecting the origin, the creation and the propagation process of data within a data-intensive system. In other words, data provenance consists in the lineage (e.g., [25]) and derivation (e.g., [21]) of data and data objects, and it puts its conceptual roots in extensively studies performed in the past in the contexts of arts, literary works, manuscripts, sculptures, and so forth. Another concept that is close to the "data provenance" one is represented by the so-called ownership of data (e.g., [20]), which refers to the issue of defining and providing information about the rightful owner of data assets, and to the acquisition, use and distribution policy implemented by the data owner.…”

Intel "big data" science and technology center vision and execution plan

“…Hence, whenever a result is calculated, a user should be able to trace the derivation of data, if he believes the result to be suspect. We have built an elaborate system that does exactly that, exploiting the semantics of relational and array operators to be able to efficiently work backwards, using a notion of fine-grained provenance [7]. We are also currently investigating visualization and other tools to help users understand data quality [20].…”