Sketching asynchronous data streams over sliding windows

Xu, Bojian; Tirthapura, Srikanta; Busch, Costas

doi:10.1007/s00446-007-0048-7

Cited by 25 publications

(43 citation statements)

References 22 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Stream sampling has a long history of research, starting from the popular reservoir sampling algorithm, attributed to Waterman (see Algorithm R from [13]) that has been known since the 1960s. Follow-up work includes speeding up reservoir sampling [13], weighted reservoir sampling [14], sampling over a sliding window and stream evolution [15], [16], [17], [18], [19]. Stream sampling has been used extensively in large scale data mining applications, see for example [20], [21], [22], [23].…”

Section: Related Workmentioning

confidence: 99%

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Chung

Tirthapura

Woodruff

2016

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Abstract-We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Keywords

show abstract

Section: Related Workmentioning

confidence: 99%

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Chung

Tirthapura

Woodruff

2016

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…To our knowledge, all previous work on computing time-decayed aggregates on streams, including [14,6,18,22,19,4,34,10,27,35], considered decomposable decay functions. For example, exponential decay, sliding window decay, and polynomial decay are all decomposable.…”

Section: Definition 13 a Decay Function F (W X) Is A Decomposable mentioning

confidence: 99%

“…This algorithm was generalized by Pavan and Tirthapura [32] to compute the duplicate insensitive sum as well as other aggregates such as max-dominance norm. Xu, Tirthapura, and Busch [34] proposed the concept of asynchronous streams and gave a randomized algorithm to approximate the sum and median over a sliding window. Here, we extend this line of work to handle both general decay and duplicate arrivals.…”

Section: Related Workmentioning

confidence: 99%

“…In the asynchronous case, it is possible that i > j, so that e i is received later than e j , but t i < t j . Most prior research on summarizing data streams containing timestamps (with the exception of [34,10]) has focused on the case of synchronous streams, where the elements of the stream are assumed to arrive in the order of timestamps. There is a growing body of work (for example, [7]) that recognizes that the assumption of synchronous streams may not always be practical and that there is a need to study more complex temporal models, such as asynchronous arrivals.…”

mentioning

confidence: 99%

“…An example decay function is the sliding window model [18,22,34], where f (w, x) is defined as follows. For some window size W , if x ≤ W , then f (w, x) = w; otherwise, f (w, x) = 0.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Time-decaying Sketches for Robust Aggregation of Sensor Data

Cormode¹,

Tirthapura²,

Xu³

2010

SIAM J. Comput.

Self Cite

View full text Add to dashboard Cite

We present a new sketch for summarizing network data. The sketch has the following properties which make it useful in communication-efficient aggregation in distributed streaming scenarios, such as sensor networks: the sketch is duplicate insensitive, i.e., reinsertions of the same data will not affect the sketch and hence the estimates of aggregates. Unlike previous duplicate-insensitive sketches 2004, pp. 449-460], it is also time decaying, so that the weight of a data item in the sketch can decrease with time according to a user-specified decay function. The sketch can give provably approximate guarantees for various aggregates of data, including the sum, median, quantiles, and frequent elements. The size of the sketch and the time taken to update it are both polylogarithmic in the size of the relevant data. Further, multiple sketches computed over distributed data can be combined without loss of accuracy. To our knowledge, this is the first sketch that combines all the above properties. [449][450][451][452][453][454][455][456][457][458][459][460], it is also time decaying, so that the weight of a data item in the sketch can decrease with time according to a user-specified decay function. The sketch can give provably approximate guarantees for various aggregates of data, including the sum, median, quantiles, and frequent elements. The size of the sketch and the time taken to update it are both polylogarithmic in the size of the relevant data. Further, multiple sketches computed over distributed data can be combined without loss of accuracy. To our knowledge, this is the first sketch that combines all the above properties.Key words. sensor network, data streams, time decay, asynchrony, data aggregation, duplicates AMS subject classifications. 68P15, 68P05, 68W20, 68W25, 68W40 DOI. 10.1137/08071795X1. Introduction. The growing size and scope of sensor networks has led to greater demand for energy-efficient communication of sensor data. Although sensors are increasing in computing ability, they remain constrained by the cost of communication, since this is the primary drain on their limited battery power. It is widely agreed that the working life of a sensor network can be extended by algorithms which limit communication [28]. In particular, this means that although sensors may observe large quantities of information over time, they should preferably return only small summaries of their observations. Ideally, we should be able to use a single compact summary that is flexible enough to provide estimates for a variety of aggregates, rather than using different summaries for estimating different aggregates.The sensor network setting leads to several other desiderata. Because of the radio network topology, it is common to take advantage of the "local broadcast" behavior, where a single transmission can be received by all the neighboring nodes. Here, in communicating back to the base station, each sensor opportunistically listens for information from other sensors, merges received information together wit...

show abstract

Time‐decayed correlated aggregates over data streams

Cormode

Tirthapura

2009

Statistical Analysis

Self Cite

View full text Add to dashboard Cite

Data stream analysis frequently relies on identifying correlations and posing conditional queries on the data after it has been seen. Correlated aggregates form an important example of such queries, which ask for an aggregation over one dimension of stream elements which satisfy a predicate on another dimension. Since recent events are typically more important than older ones, time decay should also be applied to downweight less significant values. We present space-efficient algorithms as well as space lower bounds for the time-decayed correlated sum, a problem at the heart of many related aggregations. By considering different fundamental classes of decay functions, we separate cases where efficient approximations with relative error or additive error guarantees are possible, from other cases where linear space is necessary to approximate. In particular, we show that no efficient algorithms with relative error guarantees are possible for the popular sliding window and exponential decay models, resolving an open problem. This negative result for the exponential decay holds even if the stream is allowed to be processed in multiple passes. The results are surprising, since efficient approximations are known for other data stream problems under these decay models. This is a step toward better understanding which sophisticated queries can be answered on massive streams using limited memory and computation. 

show abstract

Sketching asynchronous data streams over sliding windows

Cited by 25 publications

References 22 publications

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Time-decaying Sketches for Robust Aggregation of Sensor Data

Time‐decayed correlated aggregates over data streams

Contact Info

Product

Resources

About