Titian

Abstract

Debugging data processing logic in Data-Intensive Scalable Computing (DISC) systems is a difficult and time consuming effort. Today's DISC systems offer very little tooling for debugging programs, and as a result programmers spend countless hours collecting evidence ( e.g. , from log files) and performing trial and error debugging. To aid this effort, we built Titian , a library that enables data provenance ---tracking data through transformations---in Apache Spark. Data scientists using the Titian Spark extension will be able to quickly identify the input data at the root cause of a potential bug or outlier result. Titian is built directly into the Spark platform and offers data provenance support at interactive speeds---orders-of-magnitude faster than alternative solutions---while minimally impacting Spark job performance; observed overheads for capturing data lineage rarely exceed 30% above the baseline job execution time.

Data Provenance for Agent-Based Models in a Distributed Memory

“…Common debugging activities such as identifying intermediate data that lead to an exception, and trial-and-error task rerun with varying inputs, motivate provenance capture in DISC systems [35]. However, in the case of distributed execution in MASS, sibling task independence cannot be assumed, and it is necessary to track subtasks and internal agent state.…”

“…Triples consist of a subject, predicate and object, which represent a source node, a directed edge and a destination node, respectively. Deferred graph construction reduces provenance collection overhead at the expense of delayed query, as discussed in related literature [35,41]. While details of the ProvMASS provenance model are discussed further in Section 4.8, independent triples are mentioned here because they not only support delayed graph construction, but also enable provenance stores to be detached from dormant threads and recycled.…”

“…Provenance capture techniques for distributed and parallel computing environments have been discussed in the context of data intensive scalable computing (DISC) systems such as Hadoop [38][39][40][41] and Spark [35]. Several techniques add provenance-related identifiers to augment transformation input and propagate provenance through the shuffle stage, pairing map and reduce tasks [38,40].…”

“…However, this approach results in a larger memory footprint and corresponding performance degradation, as the data provenance size grows with each mapping. To reduce this performance degradation, some DISC-related provenance capture techniques employ framework instrumentation (i.e., adding provenance capture statements to framework source code), leveraging access to data mappings between tasks [35,41]. However, data are no longer consistently identified in situ, delaying graph construction and queries.…”

“…In addition, provenance in DISC systems assumes sibling task independence [35,38,39,41], alleviating the need to account for relationships between tasks, subtasks and intermediate data in the provenance record. Meanwhile, reasoning about operations on distributed shared resources (e.g., agents and spatial data) in the MASS environment requires provenance of these relationships.…”