Tracking of multiple quantiles in dynamically varying data streams

Abstract

In this paper we consider the problem of estimating quantiles when data are received sequentially (data stream). For real life data streams, the distribution of the data typically varies with time making estimation of quantiles challenging.We present a method that simultaneously maintain estimates of multiple quantiles of the data stream distribution. The method is based on making incremental updates of the quantile estimates every time a new sample from the data stream is received. The method is memory and computationally efficient since it only stores one value for each quantile estimate and only performs one operation per quantile estimate when a new sample is received from the data stream. The estimates are realistic in the sense that the monotone property of quantiles is satisfied in every iteration. Experiments show that the method efficiently tracks multiple quantiles and outperforms state of the art methods.

On the classification of dynamical data streams using novel “Anti-Bayesian” techniques

“…Incremental quantile estimators have been documented to yield a good performance for dynamical systems, as reported in [33,6,5]. More recently, Yazidi and Hammer suggested multiplicative incremental quantile estimators that are not only more ecient then the current state-of-the-art quantile estimators for data streams, but are also far simpler to implement [12,37]. The AB classications presented here are, therefore, based on these algorithms used to estimate the quantiles 3 .…”

“…The AB classications presented here are, therefore, based on these algorithms used to estimate the quantiles 3 . We now give a short description of the algorithms by Yazidi and Hammer [12,37].…”

“…To be more specic, within a Bayesian paradigm, if one has to classify a testing sample by resorting to a single point in the feature space from each class, the optimal Bayesian strategy would be to achieve this based on the distance (for example, Euclidean or Mahalonabis) from the corresponding means or central points in the respective distributions. In this vein, in order to deal with the challenges that pertain to data streams, a large body of studies have focused on the idea of summarizing the characteristics of a data stream by rather tracking the properties of the stream, like its distributional moments (expectation, variance, skewness, kurtosis etc) or quantiles [12,37].…”

“…Quantile estimators, that are incremental in principle, have been reported in [33,6,5]. In this paper, we will, rather, invoke the more-recently introduced estimators due to Yazidi and Hammer [12,37]. Being multiplicative incremental quantile estimators, they are not only more ecient than the current stateof-the-art quantile estimators for data streams, but are also far simpler to implement.…”

“…Naturally, for such an AB classication approach, we need to track the non-stationarity of the quantiles of the classes. To achieve this, in this paper, we develop an AB approach for the online classication of data streams by applying the ecient and robust quantile estimators developed by Yazidi and Hammer [12,37].…”