Load shedding for aggregation queries over data streams

Babcock, Brian; Datar, Mayur; Motwani, Rajeev

doi:10.1109/icde.2004.1320010

Cited by 203 publications

(239 citation statements)

References 21 publications

Supporting

Mentioning

238

Contrasting

Unclassified

Order By: Relevance

“…Several research groups have picked up the challenge of replacing the previously dominant strategy of data quality degradation, i.e., load shedding (Babcock, Datar & Motwani, 2004;Tatbul, Ç etintemel & Zdonik, 2007), with resource elasticity. Nevertheless, most of the first publications focus on an optimal resource configuration only when deploying a topology and do not consider any updates at runtime, e.g., Setty et al (2014) for pub/sub systems or Florescu & Kossmann (2009) for database systems.…”

Section: Related Workmentioning

confidence: 99%

Cost-efficient enactment of stream processing topologies

Hochreiner

Vögler

Schulte

et al. 2017

PeerJ Computer Science

View full text Add to dashboard Cite

The continuous increase of unbound streaming data poses several challenges to established data stream processing engines. One of the most important challenges is the cost-efficient enactment of stream processing topologies under changing data volume. These data volume pose different loads to stream processing systems whose resource provisioning needs to be continuously updated at runtime. First approaches already allow for resource provisioning on the level of virtual machines (VMs), but this only allows for coarse resource provisioning strategies. Based on current advances and benefits for containerized software systems, we have designed a cost-efficient resource provisioning approach and integrated it into the runtime of the Vienna ecosystem for elastic stream processing. Our resource provisioning approach aims to maximize the resource usage for VMs obtained from cloud providers. This strategy only releases processing capabilities at the end of the VMs minimal leasing duration instead of releasing them eagerly as soon as possible as it is the case for threshold-based approaches. This strategy allows us to improve the service level agreement compliance by up to 25% and a reduction for the operational cost of up to 36%. Subjects

show abstract

Section: Related Workmentioning

confidence: 99%

Cost-efficient enactment of stream processing topologies

Hochreiner

Vögler

Schulte

et al. 2017

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Node s forwards a sequence of tuples to its parent P s during a round. It numbers these sequentially, e.g., (1,2,3,4,5,6). The highest sequence number is labeled n max = 6.…”

Section: Lazaridis Et Almentioning

confidence: 99%

“…Our work complements data stream systems using load shedding [22,1]. A data stream processor often needs to ''drop'' tuples at the input of operators, if its capacity does not suffice.…”

mentioning

confidence: 99%

Fault Tolerant Evaluation of Continuous Selection Queries over Sensor Data

Lazaridis

Han

Mehrotra

et al. 2009

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

We consider the problem of evaluating continuous selection queries over sensorgenerated values in the presence of faults. Small sensors are fragile, have finite energy and memory, and communicate over a lossy medium; hence, tuples produced by them may not reach the querying node, resulting in an incomplete and ambiguous answer, as any of the non-reporting sensors may have produced a tuple which was lost. We develop a protocol, FAult Tolerant Evaluation of Continuous Selection Queries (FATE-CSQ), which guarantees a user-requested level of quality in an efficient manner. When many faults occur, this may not be achievable; in that case, we aim for the best possible answer, under the query's time constraints. FATE-CSQ is designed to be resilient to different kinds of failures. Our design decisions are based on an analytical model of different fault tolerance strategies based on feedback and retransmission. Additionally, we evaluate FATE-CSQ and competing protocols with realistic simulation parameters under a variety of conditions, demonstrating its good performance.

show abstract

“…In this case, we need to decide whether s is to be discarded or admitted into S 1 [W 1 ], and if it is to be admitted, which of the existing tuples is to be discarded. An algorithm that makes this decision is called a load-shedding strategy [4,7,17]. Due to load-shedding, only a fraction of the true result will actually be produced.…”

Section: Max-subsetmentioning

confidence: 99%

“…The stream system could instead (or also) be CPU-limited, i.e., the rate of incoming tuples is higher than can be processed. Load-shedding for the CPU-limited case has been considered in [4,17]. Sampling from a window is addressed in [3], but only for a single stream and not for a join result.…”

Section: Related Workmentioning

confidence: 99%

Memory-Limited Execution of Windowed Stream Joins

Srivastava

Widom

2004

Proceedings 2004 VLDB Conference

View full text Add to dashboard Cite

We address the problem of computing approximate answers to continuous sliding-window joins over data streams when the available memory may be insufficient to keep the entire join state. One approximation scenario is to provide a maximum subset of the result, with the objective of losing as few result tuples as possible. An alternative scenario is to provide a random sample of the join result, e.g., if the output of the join is being aggregated. We show formally that neither approximation can be addressed effectively for a slidingwindow join of arbitrary input streams. Previous work has addressed only the maximum-subset problem, and has implicitly used a frequencybased model of stream arrival. We address the sampling problem for this model. More importantly, we point out a broad class of applications for which an age-based model of stream arrival is more appropriate, and we address both approximation scenarios under this new model. Finally, for the case of multiple joins being executed with an overall memory constraint, we provide an algorithm for memory allocation across the joins that optimizes a combined measure of approximation in all scenarios considered. All of our algorithms are implemented and experimental results demonstrate their effectiveness.

show abstract

Load shedding for aggregation queries over data streams

Abstract: Abstract

Cited by 203 publications

References 21 publications

Cost-efficient enactment of stream processing topologies

Cost-efficient enactment of stream processing topologies

Fault Tolerant Evaluation of Continuous Selection Queries over Sensor Data

Memory-Limited Execution of Windowed Stream Joins

Contact Info

Product

Resources

About