Processing complex aggregate queries over data streams

Abstract

Recent years have witnessed an increasing interest in designing algorithms for querying and analyzing streaming data (i.e., data that is seen only once in a fixed order) with only limited memory. Providing (perhaps approximate) answers to queries over such continuous data streams is a crucial requirement for many application environments; examples include large telecom and IP network installations where performance data from different parts of the network needs to be continuously collected and analyzed.In this paper, we consider the problem of approximately answering general aggregate SQL queries over continuous data streams with limited memory. Our method relies on randomizing techniques that compute small "sketch" summaries of the streams that can then be used to provide approximate answers to aggregate queries with provable guarantees on the approximation error. We also demonstrate how existing statistical information on the base data (e.g., histograms) can be used in the proposed framework to improve the quality of the approximation provided by our algorithms. The key idea is to intelligently partition the domain of the underlying attribute(s) and, thus, decompose the sketching problem in a way that provably tightens our guarantees. Results of our experimental study with real-life as well as synthetic data streams indicate that sketches provide significantly more accurate answers compared to histograms for aggregate queries. This is especially true when our domain partitioning methods are employed to further boost the accuracy of the final estimates.

An efficient cube structure for stream data analysis

“…In addition to system-building efforts, a number of approximation techniques have been studied in the context of streams, such as sampling [8], sketches [7] [19], histograms [12] [13], and wavelets [14]. The main goal of 978-1-4673-1931-7/12/$31.00 ©2012 IEEE 2012 International Conference on Information Management, Innovation Management and Industrial Engineering these methods is to estimate a global aggregate (e.g.…”

Mining Loosely Structured Motifs from Biological Data

“…In recent years, several sketches have been proposed to approximate a number of simple aggregation functions (e.g., [17], [2], [23], and [18]). In fact, computing sketches has been used as a technique to speed up computation even outside the stream scenario (e.g., [31] and [15]).…”

Space-efficient Relative Error Order Sketch over Data Streams

“…Statistics computation in data streams is often required by many applications, including processing of relational type queries, [3,8,9,10,16,17], data mining [18,24,28], and high speed network management [4,6]. Among various statistics, order statistics computation is one of the most challenging, and is employed in many real applications, such as web ranking aggregation and log mining [1,12], sensor data analysis [14], trends and fleeting opportunities detection in stock markets [3,23], and load balanced data partitioning for distributed computation [25,27].…”