Effective Computation of Biased Quantiles over Data Streams

Abstract

Skew is prevalent in many data sources such as IP traffic streams. To continually summarize the distribution of such data, a high-biased set of quantiles (e.g., 50th, 90th and 99th percentiles) with finer error guarantees at higher ranks (e.g., errors of 5, 1 and 0.1 percent, respectively) is more useful than uniformly distributed quantiles (e.g., 25th, 50th and 75th percentiles) can become very stretched. Hence, to gauge the performance of the network in detail and its effect on all users (not just those experiencing the average performance), it is important to know not only the median RTT but also the 90%, 95% and 99% quantiles of TCP round trip times to each destination. In developing data stream management systems that interact with IP traffic data, there exists the facility for posing such queries [4]. However, the challenge is to develop approaches to answer such queries efficiently and accurately given that there may be many destinations to track. In such settings, the data rate is typically high and resources are limited in comparison to the amount of data that is observed. Hence it is often necessary to adopt the data stream methodology [2,6,16]: analyze IP packet headers in one pass over the data with storage space and per-packet processing time that is significantly sublinear in the size of the input.Typically, IP traffic streams and other streams are summarized using quantiles: these are order statistics such as the minimum, maximum and median values. In a data set of size n, the φ-quantile is the item with rank φn .1 The minimum and maximum are easy to calculate precisely in one pass but exact computation of certain quantiles can require space linear in n [13]. So the notion of -approximate quantiles relaxes the requirement to finding an item with rank between (φ− )n and (φ+ )n. Much attention has been given to the case of finding a set of uniform quantiles: given 0 < φ < 1, return the approximate φ, 2φ, 3φ, . . . , 1/φ φ quantiles of a stream of values.2 Note that the error in the rank of each returned value is bounded by the same amount, n; we call this the uniform error case.Summarizing distributions which have high skew using uniform quantiles is not always informative because it 1 We use the rank of an item to refer to its position in the sorted order of items that have been observed.2 While existing formal problem definitions (eg, [7]) find a single order statistic at rank φn , it is trivial to modify the output routine to return uniform quantiles as defined above; this is consistent with how "quantile" is defined in the statistics literature (eg, "percentiles" when φ = 0.01).

Norm, Point, and Distance Estimation Over Multiple Signals Using Max-Stable Distributions

“…A large body of work has appeared recently on designing approximate, probabilistic algorithms for summarizing signals on the data streaming model [17,8,6,13,16,10,11]. These algorithms focus on individual distributions, or on processing multiple distributions individually.…”

Quantile-based KNN over multi-valued objects

“…As a tool to summarize data distribution, quantile computation has been extensively studied (e.g., [9], [26]). The most related work is the quantile computation in a multidimensional space [26].…”

Continuously monitoring top-k uncertain data streams: a probabilistic threshold method

“…It has been shown in [4,25,26,44] that a space-efficient φ-approximation quantile sketch can be maintained so that, for a quantile φ, it is always possible to find an element at rank r with the uniform precision guarantee r − φN ≤ N . Due to the observation that many real data sets often exhibit skew towards heads (or tails depending on a given monotonic order), relative rank error (or biased) quantile computation techniques have been recently developed [12,13,57], which give better rank error guarantees towards heads.…”