Towards Low Overhead Provenance Tracking in Near Real-Time Stream Filtering

Vijayakumar, Nithya N.; Plale, Beth

doi:10.1007/11890850_6

Cited by 38 publications

(19 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be categorized with coarse-grained provenance methods that identify dependencies between streams or sets of streams [27,26], and fine-grained methods that identify dependencies among individual stream elements [23,10,18,22]. Sansrimahachai et al [23] propose the Stream Ancestor Function -a reverse mapping function to express precise dependencies between input and output stream elements (fine-grained).…”

Section: Related Workmentioning

confidence: 99%

Analysis of Memory Constrained Live Provenance

Chen

Plale

2016

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We conjecture that meaningful analysis of large-scale provenance can be preserved by analyzing provenance data in limited memory while the data is still in motion; that the provenance needs not be fully resident before analysis can occur. As a proof of concept, this paper defines a stream model for reasoning about provenance data in motion for Big Data provenance. We propose a novel streaming algorithm for the backward provenance query, and apply it to the live provenance captured from agent-based simulations. The performance test demonstrates high throughput, low latency and good scalability, in a distributed stream processing framework built on Apache Kafka and Spark Streaming.

show abstract

Section: Related Workmentioning

confidence: 99%

Analysis of Memory Constrained Live Provenance

Chen

Plale

2016

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…They incorporated both accuracy and provenance as an integrated part of data management and query processing. Vijayakkumar and Plale [32] introduced the provenance collection framework for an event processing system. Sarma et al [33] developed an architecture based on a decoupled strategy to compute provenance and confidence in probabilistic databases.…”

Section: History Of Trustworthiness Frameworkmentioning

confidence: 99%

“…First, Vijayakumar and Plale [32], proposed a provenance collection framework for collecting provenance with low overhead in stream filtering applications. They addressed the challenges of the data streaming environment and proposed data collection models with low overhead for handling rapid streams.…”

Section: Some Other Prominent Frameworkmentioning

confidence: 99%

Comparative Study of Evaluating the Trustworthiness of Data Based on Data Provenance

2016

J Inf Process Syst

View full text Add to dashboard Cite

Due to the proliferation of data being exchanged and the increase of dependency on this data for critical decision-making, it has become imperative to ensure the trustworthiness of the data at the receiving end in order to obtain reliable results. Data provenance, the derivation history of data, is a useful tool for evaluating the trustworthiness of data. Various frameworks have been proposed to evaluate the trustworthiness of data based on data provenance. In this paper, we briefly review a history of these frameworks for evaluating the trustworthiness of data and present an overview of some prominent state-of-the-art evaluation frameworks. Moreover, we provide a comparative analysis of two key frameworks by evaluating various aspects in an executional environment. Our analysis points to various open research issues and provides an understanding of the functionalities of the frameworks that are used to evaluate the trustworthiness of data.

show abstract

“…Sensors and data streaming techniques are increasingly used in a wide range of applications, from science (weather forecast [393]) to health care (remote health monitoring [287]). Such streamed data are used by sophisticated simulation, modelling and analysis tools.…”

Section: Data Collections and Streamsmentioning

confidence: 99%

“…Vijayakumar and Plale [393,392] adopt a stream as the unit of data they track provenance for, so as to ensure a lightweight and low overhead approach to provenance recording in distributed stream applications. The downside is that the granularity of their provenance model is so coarse that it is unable to answer queries related to individual stream elements.…”

Section: Data Collections and Streamsmentioning

confidence: 99%

The Foundations for Provenance on the Web

Moreau

2010

FNT in Web Science

168

View full text Add to dashboard Cite

Provenance, i.e., the origin or source of something, is becoming an important concern, since it offers the means to verify data products, to infer their quality, to analyse the processes that led to them, and to decide whether they can be trusted. For instance, provenance enables the reproducibility of scientific results; provenance is necessary to track attribution and credit in curated databases; and, it is essential for reasoners to make trust judgements about the information they use over the Semantic Web.As the Web allows information sharing, discovery, aggregation, filtering and flow in an unprecedented manner, it also becomes very difficult to identify, reliably, the original source that produced an information item on the Web. Since the emerging use of provenance in niche applications is undoubtedly demonstrating the benefits of provenance, we contend that provenance can and should reliably be tracked and exploited on the Web, and we survey the necessary foundations to achieve such a vision.Using multiple data sources, we have compiled the largest bibliographical database on provenance so far. This large corpus allows us to analyse emerging trends in the research community. Specifically, using the CiteSpace tool, we identify clusters of papers that constitute research fronts, from which we derive characteristics that we use to structure our foundational framework for provenance on the Web. We note that such an endeavour requires a multi-disciplinary approach, since it requires contributions from many computer science sub-disciplines, but also other non-technical fields given the human challenge that is anticipated.To develop our vision, it is necessary to provide a definition of provenance that applies to the Web context. Our conceptual definition of provenance is expressed in terms of processes, and is shown to generalise various definitions of provenance commonly encountered. Furthermore, by bringing realistic distributed systems assumptions, we refine our definition as a query over assertions made by processes.Given that the majority of work on provenance has been undertaken by the database, workflow and e-science communities, we review some of their work, contrasting approaches, and focusing on important topics we believe to be crucial for bringing provenance to the Web, such as abstraction, collections, storage, queries, workflow evolution, semantics and activities involving human interactions.However, provenance approaches developed in the context of databases and workflows essentially deal with closed systems. By that, we mean that workflow or database management systems are in full control of the data they manage, and track their provenance within their own scope, but not beyond. In the context of the Web, a broader approach is required by which chunks of provenance representation can be brought together to describe the provenance of information flowing across multiple systems. This is the specific purpose of the Open Provenance Vision, which is an approach that consists of controlled voc...

show abstract

Towards Low Overhead Provenance Tracking in Near Real-Time Stream Filtering

Cited by 38 publications

References 14 publications

Analysis of Memory Constrained Live Provenance

Analysis of Memory Constrained Live Provenance

Comparative Study of Evaluating the Trustworthiness of Data Based on Data Provenance

The Foundations for Provenance on the Web

Contact Info

Product

Resources

About