Estimating quantiles from the union of historical and streaming data

Abstract

Modern enterprises generate huge amounts of streaming data, for example, micro-blog feeds, financial data, network monitoring and industrial application monitoring. While Data Stream Management Systems have proven successful in providing support for real-time alerting, many applications, such as network monitoring for intrusion detection and real-time bidding, require complex analytics over historical and real-time data over the data streams. We present a new method to process one of the most fundamental analytical primitives, quantile queries, on the union of historical and streaming data. Our method combines an index on historical data with a memory-efficient sketch on streaming data to answer quantile queries with accuracy-resource tradeoffs that are significantly better than current solutions that are based solely on disk-resident indexes or solely on streaming algorithms. Disciplines Electrical and Computer Engineering CommentsThis is a manuscript of a proceeding published as Singh, Sneha Aman, Divesh Srivastava, and Srikanta Tirthapura. "Estimating quantiles from the union of historical and streaming data. Systems have proven successful in providing support for real-time alerting, many applications, such as network monitoring for intrusion detection and real-time bidding, require complex analytics over historical and real-time data over the data streams. We present a new method to process one of the most fundamental analytical primitives, quantile queries, on the union of historical and streaming data. Our method combines an index on historical data with a memory-efficient sketch on streaming data to answer quantile queries with accuracy-resource tradeoffs that are significantly better than current solutions that are based solely on disk-resident indexes or solely on streaming algorithms.

FlexSketch: Estimation of Probability Density for Stationary and Non-Stationary Data Streams

“…There also exist methods to estimate the distribution of a data stream based on histograms, including the streaming parallel decision tree (denoted by in this paper) [ 8 ], variations of the V-optimal histogram algorithm [ 40 , 41 ] and quantile summarization algorithms [ 42 , 43 , 44 , 45 ]. However, they cannot forget the contribution of the past data and adapt to various types of concept drift.…”

Online Computing Quantile Summaries Over Uncertain Data Streams

“…To reduce space requirement, many works have focused on computing approximate quantile summary and revealed that, to efficiently support various applications, the algorithms for computing such summary should have some common properties [3], [4]: (1) the algorithm should support to turn the bound on the error of the approximation; (2) the algorithm should not be implemented by the distributions of input datasets; (3) the algorithm should scan a dataset only once; (4) the algorithm should utilize as small space as possible. Many such methods have been presented [3], [4], [51]- [54]. Among them, the well-known Greenwald and Khanna [4] (GK) algorithm requires space O(log( n)/ ) for summarizing a dataset of size n and a given .…”