Fast Flow Volume Estimation

Basat, Ran Ben; Einziger, Gil; Friedman, Roy

doi:10.1145/3154273.3154332

Cited by 17 publications

(14 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, we can solve the interval volume estimation and (weighted) heavy hitters problems with the same asymptotic complexity as the unweighted variants and with an error of at most W M . This is a generalization of the result of [7] that finds weighted heavy hitters over fixed size windows.…”

Section: Supporting Traffic Volume Heavy-hittersmentioning

confidence: 80%

“…The Space Saving algorithm [30] can find weighted heavy hitters over a stream with O(log −1 ) update time [12]. Recent breakthroughs [9,7,3] improve this runtime to a constant. Thus, we can solve the interval volume estimation and (weighted) heavy hitters problems with the same asymptotic complexity as the unweighted variants and with an error of at most W M .…”

Section: Supporting Traffic Volume Heavy-hittersmentioning

confidence: 99%

See 1 more Smart Citation

Stream frequency over interval queries

Basat

Friedman²,

Shahout³

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Stream frequency measurements are fundamental in many data stream applications such as financial data trackers, intrusion-detection systems, and network monitoring. Typically, recent data items are more relevant than old ones, a notion we can capture through a sliding window abstraction. This paper considers a generalized sliding window model that supports stream frequency queries over an interval given at query time. This enables drill-down queries, in which we can examine the behavior of the system in finer and finer granularities. For this model, we asymptotically improve the space bounds of existing work, reduce the update and query time to a constant, and provide deterministic solutions. When evaluated over real Internet packet traces, our fastest algorithm processes items 90-250 times faster, serves queries at least 730 times quicker and consumes at least 40% less space than the best known method. IntroductionHigh-performance stream processing is essential for many applications such as financial data trackers, intrusiondetection systems, network monitoring, and sensor networks. Such applications require algorithms that are both time and space efficient to cope with high-speed data streams. Space efficiency is needed, due to the memory hierarchy structure, to enable cache residency and to avoid page swapping. This residency is vital for obtaining good performance, even when the theoretical computational cost is small (e.g., constant time algorithms may be inefficient if they access the DRAM for each element). To that end, stream processing algorithms often build compact approximate sketches (synopses) of the input streams.Recent items are often more relevant than old ones, which requires an aging mechanism for the sketches. Many applications realize this by tracking the stream's items over a sliding window. That is, the sliding window model [18] considers only a window of the most recent items in the stream, while older ones do not affect the quantity we wish to estimate. Indeed, the problem of maintaining different types of sliding window statistics was extensively studied [4,8,18,33,27].Yet, sometimes the window of interest may not be known a priori or they may be multiple interesting windows [17]. Further, the ability to perform drill-down queries, in which we examine the behavior of the system in finer and finer granularity may also be beneficial, especially for security applications. For example, this enables detecting when precisely a particular anomaly has started and who was involved in it [20]. Additional applications for this capability include identifying the sources of flash crowd effects and pinpointing the cause-effect relation surrounding a surge in demand on an e-commerce website [26].In this work, we study a model that allows the user to specify an interval of interest at query time. This extends traditional sliding windows that only consider fixed sized windows. As depicted in Figure 1, a sub-interval of a maximal window is passed as a parameter for each query, and the goal of the algorithm...

show abstract

Section: Supporting Traffic Volume Heavy-hittersmentioning

confidence: 80%

Section: Supporting Traffic Volume Heavy-hittersmentioning

confidence: 99%

Stream frequency over interval queries

Basat

Friedman²,

Shahout³

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Space Saving algorithm [37] can find weighted heavy hitters in a stream with an update time of 𝑂 (log 𝜖 −1 ) [14]. Recent advancements [6,9,12] reduce this runtime to a constant. Thus, the tail latency problem for weighted heavy hitters may be solved with the same asymptotic complexity as the unweighted versions and with an error of up to 𝑀𝜖.…”

Section: Extensions Of Supporting Tail Latencies For Traffic Volume H...mentioning

confidence: 99%

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

Shahout¹,

Friedman²,

Basat³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Stream monitoring is fundamental in many data stream applications, such as financial data trackers, security, anomaly detection, and load balancing. In that respect, quantiles are of particular interest, as they often capture the user's utility. For example, if a video connection has high tail latency, the perceived quality will suffer, even if the average and median latencies are low.In this work, we consider the problem of approximating the per-item quantiles. Elements in our stream are (ID, latency) tuples, and we wish to track the latency quantiles for each ID. Existing quantile sketches are designed for a single number stream (e.g., containing just the latency). While one could allocate a separate sketch instance for each ID, this may require an infeasible amount of memory. Instead, we consider tracking the quantiles for the heavy hitters (most frequent items), which are often considered particularly important, without knowing them beforehand.We first present a simple sampling algorithm that serves as a benchmark. Then, we design an algorithm that augments a quantile sketch within each entry of a heavy hitter algorithm, resulting in similar space complexity but with a deterministic error guarantee. Finally, we present SQUAD, a method that combines sampling and sketching while improving the asymptotic space complexity. Intuitively, SQUAD uses a background sampling process to capture the behaviour of the latencies of an item before it is allocated with a sketch, thereby allowing us to use fewer samples and sketches. Our solutions are rigorously analyzed, and we demonstrate the superiority of our approach using extensive simulations.

show abstract

“…For performance, this upper bound is often rounded up to be a multiple of the word size. For example, practitioners often allocate 32-bit counters when estimating the unit-count of elements, and 64-bit counters for measuring their weighted-frequency (e.g., [32], [33]). When space is tight, estimators are sometimes integrated into sketches to allow smaller (e.g., 16-bit) per-counter overhead at the cost of additional error [16].…”

Section: Techniquesmentioning

confidence: 99%

“…The SALSA encoding: SALSA starts with all counters having s bits (e.g., s = 8), where s may be significantly smaller than the intended counting range (e.g., N = 2 32 ). Here, we describe an encoding that requires one bit of overhead per counter (e.g., 12.5% for s = 8 bit counters); we later explain how to reduce it to less than 0.6 bits (7.5% for s = 8).…”

Section: Techniquesmentioning

confidence: 99%

SALSA: Self-Adjusting Lean Streaming Analytics

Basat¹,

Einziger²,

Mitzenmacher³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Counters are the fundamental building block of many data sketching schemes, which hash items to a small number of counters and account for collisions to provide good approximations for frequencies and other measures. Most existing methods rely on fixed-size counters, which may be wasteful in terms of space, as counters must be large enough to eliminate any risk of overflow. Instead, some solutions use small, fixed-size counters that may overflow into secondary structures.This paper takes a different approach. We propose a simple and general method called SALSA for dynamic re-sizing of counters, and show its effectiveness. SALSA starts with small counters, and overflowing counters simply merge with their neighbors. SALSA can thereby allow more counters for a given space, expanding them as necessary to represent large numbers. Our evaluation demonstrates that, at the cost of a small overhead for its merging logic, SALSA significantly improves the accuracy of popular schemes (such as Count-Min Sketch and Count Sketch) over a variety of tasks. Our code is released as open source [1].

show abstract

Fast Flow Volume Estimation

Cited by 17 publications

References 45 publications

Stream frequency over interval queries

Stream frequency over interval queries

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

SALSA: Self-Adjusting Lean Streaming Analytics

Contact Info

Product

Resources

About