Space-efficient Relative Error Order Sketch over Data Streams

Abstract

We consider the problem of continuously maintaining order sketches over data streams with a relative rank error guarantee . Novel space-efficient and one-scan randomised techniques are developed. Our first randomised algorithm can guarantee such a relative error precision with confidence 1 − δ using O( 1 2 log 1 δ log 2 N ) space, where N is the number of data elements seen so far in a data stream. Then, a new one-scan space compression technique is developed. Combined with the first randomised algorithm, the one-scan space compression technique yields another one-scan randomised algorithm that guarantees the space requirement is O( 1 log( 1 log 1 δ ) log 2+α N 1−1/2 α ) (for α > 0) on average while the worst case space remains O( 1 2 log 1 δ log 2 N ). These results are immediately applicable to approximately computing quantiles over data streams with a relative error guarantee and significantly improve the previous best space bound O( 1 3 log 1 δ log N ). Our extensive experiment results demonstrate that both techniques can support an on-line computation against high speed data streams.

Relative Error Streaming Quantiles

“…Each buffer of size 𝐵 "protects" the 𝐵/2 smallest items stored inside, meaning that these items are not involved in any compaction (i.e., the compaction operation only removes the 𝐵/2 largest items from the buffer). Unfortunately, it turns out that this simple approach requires space Θ(𝜀 −2 • log(𝜀 2 𝑛)), which merely matches the space bound achieved in [24], and in particular has a (quadratically) suboptimal dependence on 1/𝜀.…”

“…Gupta and Zane [11] gave an algorithm for relative error quantiles that stores 𝑂 (𝜀 −3 • log 2 (𝜀𝑛)) items, and use this to approximately count the number of inversions in a list; their algorithm requires prior knowledge of the stream length 𝑛. As previously mentioned, Zhang et al [24] presented an algorithm storing 𝑂 (𝜀 −2 • log(𝜀 2 𝑛)) universe items. Cormode et al [5] designed a deterministic sketch storing 𝑂 (𝜀 −1 • log(𝜀𝑛) • log |U|) items, which requires prior knowledge of the data universe U.…”

“…That is, their algorithm aims to provide multiplicative guarantees only up to some minimum rank 𝑘; for items of rank below 𝑘, their solution only provides additive guarantees. Their algorithm does not solve the relative error quantiles problem: [24] observed that for adversarial item ordering, the algorithm of [4] requires linear space to achieve relative error for all ranks. Dunning and Ertl [7] describe a heuristic algorithm called t-digest that is intended to achieve relative error, but they provide no formal accuracy analysis.…”

“…1 The best known algorithms achieving multiplicative error guarantees are as follows. Zhang et al [24] give a randomized algorithm storing 𝑂 (𝜀 −2 • log(𝜀 2 𝑛)) universe items. This is essentially a 𝜀 −1 factor away from the aforementioned lower bound.…”

OneSketch: A Generic and Accurate Sketch for Data Streams

“…It can be used for packet loss detection in networks, but has rarely been formally studied. Sketch, a compact data structure with small memory footprint and error, has been widely recognized by the research community [18]- [21], especially in addressing the above tasks 1 to 4. Thus, our design goal is to propose a generic sketch algorithm that can more accurately perform the above five tasks.…”