A high-performance algorithm for identifying frequent items in data streams

Anderson, D. Mark; Bevan, Pryce; Lang, Kévin; Liberty, Edo; Rhodes, Lee; Thaler, Justin

doi:10.1145/3131365.3131407

Cited by 27 publications

(9 citation statements)

References 47 publications

(83 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Space Saving algorithm [37] can find weighted heavy hitters in a stream with an update time of 𝑂 (log 𝜖 −1 ) [14]. Recent advancements [6,9,12] reduce this runtime to a constant. Thus, the tail latency problem for weighted heavy hitters may be solved with the same asymptotic complexity as the unweighted versions and with an error of up to 𝑀𝜖.…”

Section: Extensions Of Supporting Tail Latencies For Traffic Volume H...mentioning

confidence: 99%

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

Shahout¹,

Friedman²,

Basat³

2022

Preprint

View full text Add to dashboard Cite

Stream monitoring is fundamental in many data stream applications, such as financial data trackers, security, anomaly detection, and load balancing. In that respect, quantiles are of particular interest, as they often capture the user's utility. For example, if a video connection has high tail latency, the perceived quality will suffer, even if the average and median latencies are low.In this work, we consider the problem of approximating the per-item quantiles. Elements in our stream are (ID, latency) tuples, and we wish to track the latency quantiles for each ID. Existing quantile sketches are designed for a single number stream (e.g., containing just the latency). While one could allocate a separate sketch instance for each ID, this may require an infeasible amount of memory. Instead, we consider tracking the quantiles for the heavy hitters (most frequent items), which are often considered particularly important, without knowing them beforehand.We first present a simple sampling algorithm that serves as a benchmark. Then, we design an algorithm that augments a quantile sketch within each entry of a heavy hitter algorithm, resulting in similar space complexity but with a deterministic error guarantee. Finally, we present SQUAD, a method that combines sampling and sketching while improving the asymptotic space complexity. Intuitively, SQUAD uses a background sampling process to capture the behaviour of the latencies of an item before it is allocated with a sketch, thereby allowing us to use fewer samples and sketches. Our solutions are rigorously analyzed, and we demonstrate the superiority of our approach using extensive simulations.

show abstract

Section: Extensions Of Supporting Tail Latencies For Traffic Volume H...mentioning

confidence: 99%

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

Shahout¹,

Friedman²,

Basat³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The scientific work described in 19 papers is categorized as measurements [4,7,8,17,19,[25][26][27][28][29][30][31][32][33][34][35][36][37][38]. Finally, 8 papers are classified as miscellaneous [9,11,14,18,20,22,23,42].…”

Section: Grouping Of Artifactsmentioning

confidence: 99%

A Survey on Artifacts from CoNEXT, ICN, IMC, and SIGCOMM Conferences in 2017

Flittner

Mahfoudi

Saucez

et al. 2018

SIGCOMM Comput. Commun. Rev.

View full text Add to dashboard Cite

Reproducibility of artifacts is a cornerstone of most scientific publications. To improve the current state and strengthen ongoing community efforts towards reproducibility by design, we conducted a survey among the papers published at leading ACM computer networking conferences in 2017: CoNEXT, ICN, IMC, and SIGCOMM. The objective of this paper is to assess the current state of artifact availability and reproducibility based on a survey. We hope that it will serve as a starting point for further discussions to encourage researchers to ease the reproduction of scientific work published within the SIGCOMM community. Furthermore, we hope this work will inspire program chairs of future conferences to emphasize reproducibility within the ACM SIGCOMM community as well as will strengthen awareness of researchers.

show abstract

“…The Space Saving algorithm [30] can find weighted heavy hitters over a stream with O(log −1 ) update time [12]. Recent breakthroughs [9,7,3] improve this runtime to a constant. Thus, we can solve the interval volume estimation and (weighted) heavy hitters problems with the same asymptotic complexity as the unweighted variants and with an error of at most W M .…”

Section: Supporting Traffic Volume Heavy-hittersmentioning

confidence: 99%

Stream frequency over interval queries

Basat

Friedman²,

Shahout³

2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

Stream frequency measurements are fundamental in many data stream applications such as financial data trackers, intrusion-detection systems, and network monitoring. Typically, recent data items are more relevant than old ones, a notion we can capture through a sliding window abstraction. This paper considers a generalized sliding window model that supports stream frequency queries over an interval given at query time. This enables drill-down queries, in which we can examine the behavior of the system in finer and finer granularities. For this model, we asymptotically improve the space bounds of existing work, reduce the update and query time to a constant, and provide deterministic solutions. When evaluated over real Internet packet traces, our fastest algorithm processes items 90-250 times faster, serves queries at least 730 times quicker and consumes at least 40% less space than the best known method. IntroductionHigh-performance stream processing is essential for many applications such as financial data trackers, intrusiondetection systems, network monitoring, and sensor networks. Such applications require algorithms that are both time and space efficient to cope with high-speed data streams. Space efficiency is needed, due to the memory hierarchy structure, to enable cache residency and to avoid page swapping. This residency is vital for obtaining good performance, even when the theoretical computational cost is small (e.g., constant time algorithms may be inefficient if they access the DRAM for each element). To that end, stream processing algorithms often build compact approximate sketches (synopses) of the input streams.Recent items are often more relevant than old ones, which requires an aging mechanism for the sketches. Many applications realize this by tracking the stream's items over a sliding window. That is, the sliding window model [18] considers only a window of the most recent items in the stream, while older ones do not affect the quantity we wish to estimate. Indeed, the problem of maintaining different types of sliding window statistics was extensively studied [4,8,18,33,27].Yet, sometimes the window of interest may not be known a priori or they may be multiple interesting windows [17]. Further, the ability to perform drill-down queries, in which we examine the behavior of the system in finer and finer granularity may also be beneficial, especially for security applications. For example, this enables detecting when precisely a particular anomaly has started and who was involved in it [20]. Additional applications for this capability include identifying the sources of flash crowd effects and pinpointing the cause-effect relation surrounding a surge in demand on an e-commerce website [26].In this work, we study a model that allows the user to specify an interval of interest at query time. This extends traditional sliding windows that only consider fixed sized windows. As depicted in Figure 1, a sub-interval of a maximal window is passed as a parameter for each query, and the goal of the algorithm...

show abstract

A high-performance algorithm for identifying frequent items in data streams

Cited by 27 publications

References 47 publications

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

A Survey on Artifacts from CoNEXT, ICN, IMC, and SIGCOMM Conferences in 2017

Stream frequency over interval queries

Contact Info

Product

Resources

About