General incremental sliding-window aggregation

Tangwongsan, Kanat; Hirzel, Martin; Schneider, Scott; Wu, Kun-Lung

doi:10.14778/2752939.2752940

Cited by 86 publications

(62 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For faster answering median like function, which has to keep all the relevant inputs, instead of performing a sort on the window for each newly inserted tuple, one can maintain an order statistics tree as auxiliary data structure [34], which has O(logn) worst-case complexity of its insertion, deletion, and rank function. Similarly, the reactive aggregator (RA) [80] with O(logn) average complexity only works for aggregation function with the associative property. Those algorithms also differ from each other at their capability of handling different window types, windowing measures, and stream (dis)order [86].…”

Section: Window Aggregationmentioning

confidence: 99%

Hardware-Conscious Stream Processing

Zhang

et al. 2020

SIGMOD Rec.

View full text Add to dashboard Cite

Data stream processing systems (DSPSs) enable users to express and run stream applications to continuously process data streams. To achieve realtime data analytics, recent researches keep focusing on optimizing the system latency and throughput. Witnessing the recent great achievements in the computer architecture community, researchers and practitioners have investigated the potential of adoption hardware-conscious stream processing by better utilizing modern hardware capacity in DSPSs. In this paper, we conduct a systematic survey of recent work in the field, particularly along with the following three directions: 1) computation optimization, 2) stream I/O optimization, and 3) query deployment. Finally, we advise on potential future research directions.

show abstract

Section: Window Aggregationmentioning

confidence: 99%

Hardware-Conscious Stream Processing

Zhang

et al. 2020

SIGMOD Rec.

View full text Add to dashboard Cite

show abstract

“…The De-Amortized Bankers Aggregator (DABA) also only works on in-order data and is worst-case O(1) [28]. The Reactive Aggregator supports out-of-order evict but requires in-order insert and is amortized O(log n) with worstcase O(n) [29]. The x-axis represents increasing window size n.…”

Section: Fifomentioning

confidence: 99%

Optimal and general out-of-order sliding-window aggregation

Tangwongsan

Hirzel²,

Schneider³

2019

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Sliding-window aggregation summarizes the most recent information in a data stream. Users specify how that summary is computed, usually as an associative binary operator because this is the most general known form for which it is possible to avoid naïvely scanning every window. For strictly in-order arrivals, there are algorithms with O(1) time per window change assuming associative operators. Meanwhile, it is common in practice for streams to have data arriving slightly out of order, for instance, due to clock drifts or communication delays. Unfortunately, for out-of-order streams, one has to resort to latency-prone buffering or pay O(log n) time per insert or evict, where n is the window size.This paper presents the design, analysis, and implementation of FiBA, a novel sliding-window aggregation algorithm with an amortized upper bound of O(log d) time per insert or evict, where d is the distance of the inserted or evicted value to the closer end of the window. This means O(1) time for in-order arrivals and nearly O(1) time for slightly outof-order arrivals, with a smooth transition towards O(log n) as d approaches n. We also prove a matching lower bound on running time, showing optimality. Our algorithm is as general as the prior state-of-the-art: it requires associativity, but not invertibility nor commutativity. At the heart of the algorithm is a careful combination of finger-searching techniques, lazy rebalancing, and position-aware partial aggregates. We further show how to answer range queries that aggregate subwindows for window sharing. Finally, our experimental evaluation shows that FiBA performs well in practice and supports the theoretical findings.

show abstract

“…In particular, we allow plugging in arbitrary code for handling the insertion of new elements in the window and the eviction of expiring elements. So, all algorithmic techniques proposed in the literature for sliding-window aggregation [16, 41, 42] can be seamlessly integrated in StreamQRE. The main point here is that efficient sliding-window aggregation algorithms can be nested arbitrarily with the regular constructs of StreamQRE without incurring any additional computational overhead.…”

Section: The Streamqre Languagementioning

confidence: 99%

StreamQRE: modular specification and efficient evaluation of quantitative queries over streaming data

et al. 2017

View full text Add to dashboard Cite

Real-time decision making in emerging IoT applications typically relies on computing quantitative summaries of large data streams in an efficient and incremental manner. To simplify the task of programming the desired logic, we propose StreamQRE, which provides natural and high-level constructs for processing streaming data. Our language has a novel integration of linguistic constructs from two distinct programming paradigms: streaming extensions of relational query languages and quantitative extensions of regular expressions. The former allows the programmer to employ relational constructs to partition the input data by keys and to integrate data streams from different sources, while the latter can be used to exploit the logical hierarchy in the input stream for modular specifications. We first present the core language with a small set of combinators, formal semantics, and a decidable type system. We then show how to express a number of common patterns with illustrative examples. Our compilation algorithm translates the high-level query into a streaming algorithm with precise complexity bounds on per-item processing time and total memory footprint. We also show how to integrate approximation algorithms into our framework. We report on an implementation in Java, and evaluate it with respect to existing high-performance engines for processing streaming data. Our experimental evaluation shows that (1) StreamQRE allows more natural and succinct specification of queries compared to existing frameworks, (2) the throughput of our implementation is higher than comparable systems (for example, two-to-four times greater than RxJava), and (3) the approximation algorithms supported by our implementation can lead to substantial memory savings.

show abstract

General incremental sliding-window aggregation

Cited by 86 publications

References 37 publications

Hardware-Conscious Stream Processing

Hardware-Conscious Stream Processing

Optimal and general out-of-order sliding-window aggregation

StreamQRE: modular specification and efficient evaluation of quantitative queries over streaming data

Contact Info

Product

Resources

About