Visual Debugging for Stream Processing Applications

Pauw, Wim De; Leţia, Mihai; Gedik, Buğra; Andrade, Henrique; Frenkiel, Andy; Pfeifer, Michael; Sow, Daby

doi:10.1007/978-3-642-16612-9_3

Cited by 18 publications

(23 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another previous work [14], proposes the recording of provenance data only for portions of the workflow, supporting the argument that the generation of huge quantity of trace data can be excessively demanding for a visualisation tool. Moreover, it can turn into a cognitive overload for the analysis of a realistic application.…”

Section: Related Workmentioning

confidence: 99%

Provenance for seismological processing pipelines in a distributed streaming workflow

Spinuso

Cheney

Atkinson

2013

Proceedings of the Joint EDBT/ICDT 2013 Workshops

View full text Add to dashboard Cite

Harvesting provenance for streaming workflows presents challenges related to the high rate of the updates and a large distribution of the execution, which can be spread across several institutional infrastructures. Moreover, the typically large volume of data produced by each transformation step can not be always stored and preserved efficiently. This can represent an obstacle for the evaluation of the results, for instance, in real-time, suggesting the importance of customisable metadata extraction procedures. In this paper we present our approach to the aforementioned provenance challenges within a use-case driven scenario in the field of seismology, which requires the execution of processing pipelines over a large datastream. In particular, we will discuss the current implementation and the upcoming challenges for an in-worfklow programmatic approach to provenance tracing, building on composite functions, selective recording and domain specific metadata production.

show abstract

Section: Related Workmentioning

confidence: 99%

Provenance for seismological processing pipelines in a distributed streaming workflow

Spinuso

Cheney

Atkinson

2013

Proceedings of the Joint EDBT/ICDT 2013 Workshops

View full text Add to dashboard Cite

show abstract

“…It can be categorized with coarse-grained provenance methods that identify dependencies between streams or sets of streams [27,26], and fine-grained methods that identify dependencies among individual stream elements [23,10,18,22]. Sansrimahachai et al [23] propose the Stream Ancestor Function -a reverse mapping function to express precise dependencies between input and output stream elements (fine-grained).…”

Section: Related Workmentioning

confidence: 99%

Analysis of Memory Constrained Live Provenance

Chen

Plale

2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We conjecture that meaningful analysis of large-scale provenance can be preserved by analyzing provenance data in limited memory while the data is still in motion; that the provenance needs not be fully resident before analysis can occur. As a proof of concept, this paper defines a stream model for reasoning about provenance data in motion for Big Data provenance. We propose a novel streaming algorithm for the backward provenance query, and apply it to the live provenance captured from agent-based simulations. The performance test demonstrates high throughput, low latency and good scalability, in a distributed stream processing framework built on Apache Kafka and Spark Streaming.

show abstract

“…Thus, most issues caused by non-determinism are dealt with in a rather natural way, since the execution of the original query network is traced 2 . Provenance can be traced for a single operator (as supported by previous approaches [12]) or for a complete subnetwork. Furthermore, we can trace provenance for a subnetwork by instrumenting only operators in that subnetwork.…”

Section: The Operator Instrumentation Approachmentioning

confidence: 99%

“…This information can be used to track coarse-grained provenance. The visual debugger proposed in [12] supports finegrained provenance computation based on identifier annotation and operator instrumentation. Our approach is more general in that we support multi-step provenance, can decouple provenance computation from regular query processing, and compress provenance.…”

Section: Related Workmentioning

confidence: 99%

“…This approach can also be used to compute the provenance of a part of the query network by only instrumenting a subset of the operators. Previous annotation propagation approaches for fine-grained stream provenance [12] are restricted to one-step provenance, i.e., annotating output tuples from an operator with their provenance from the operator's input. Our approach is more general and flexible; provenance can also be propagated through several operators or even a complete query network.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ariadne

Glavic

Esmaili

Fischer

et al. 2013

Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems

View full text Add to dashboard Cite

Managing fine-grained provenance is a critical requirement for data stream management systems (DSMS), not only to address complex applications that require diagnostic capabilities and assurance, but also for providing advanced functionality such as revision processing or query debugging. This paper introduces a novel approach that uses operator instrumentation, i.e., modifying the behavior of operators, to generate and propagate fine-grained provenance through several operators of a query network. In addition to applying this technique to compute provenance eagerly during query execution, we also study how to decouple provenance computation from query processing to reduce run-time overhead and avoid unnecessary provenance retrieval. This includes computing a concise superset of the provenance to allow lazily replaying a query network and reconstruct its provenance as well as lazy retrieval to avoid unnecessary reconstruction of provenance. We develop stream-specific compression methods to reduce the computational and storage overhead of provenance generation and retrieval. Ariadne, our provenance-aware extension of the Borealis DSMS implements these techniques. Our experiments confirm that Ariadne manages provenance with minor overhead and clearly outperforms query rewrite, the current state-of-the-art.

show abstract

Visual Debugging for Stream Processing Applications

Cited by 18 publications

References 14 publications

Provenance for seismological processing pipelines in a distributed streaming workflow

Provenance for seismological processing pipelines in a distributed streaming workflow

Analysis of Memory Constrained Live Provenance

Ariadne

Contact Info

Product

Resources

About