Analysis of Memory Constrained Live Provenance

Abstract

Abstract. We conjecture that meaningful analysis of large-scale provenance can be preserved by analyzing provenance data in limited memory while the data is still in motion; that the provenance needs not be fully resident before analysis can occur. As a proof of concept, this paper defines a stream model for reasoning about provenance data in motion for Big Data provenance. We propose a novel streaming algorithm for the backward provenance query, and apply it to the live provenance captured from agent-based simulations. The performance test demonstrates high throughput, low latency and good scalability, in a distributed stream processing framework built on Apache Kafka and Spark Streaming.

Big Provenance Stream Processing for Data Intensive Computations

“…To best of our knowledge, the only attempt in the literature to use stream processing techniques for analyzing a stream of provenance is made by Chen, P. et al [28] where the authors compute a dependency matrix between the input parameters and the variable values in an Agent-Based simulation using a stream of provenance. However, there are multiple drawbacks in their work.…”

“…Finally, we evaluate the query performance for backward and forward provenance on full provenance vs reduced provenance. In the first experiment, we measure the accuracy of our parallel-prov-stream algorithm for out-of-order provenance streams and compare results with Chen et al [28]. Accuracy is measured by calculating the percentage of correct backward and forward provenance relationships exist in the output provenance graph.…”

“…Accuracy is measured by calculating the percentage of correct backward and forward provenance relationships exist in the output provenance graph. The algorithm in [28] is not designed to handle out-of-order events. Therefore, the intention of this experiment is not to show that our algorithm performs better.…”

“…Closest to our research is Chen and Plale [28] where a dependency matrix is computed across input parameters and variable values from a stream of data provenance from an Agent-Based model. Our work here both extends and complements Chen and Plale through application of provenance stream processing to large scale DICs.…”

Data Provenance for Agent-Based Models in a Distributed Memory

“…In situ queries can be evaluated as elements of a provenance stream are generated and subsequently discarded [50]. This reduces space requirements for provenance-based big data analysis.…”