CVS: Fast cardinality estimation for large-scale data streams over sliding windows

Abstract

Estimating the cardinality of data streams over a sliding window is an important problem in many applications, such as network traffic monitoring, web access log analysis and database. The problem becomes more difficult in largescale data streams when time and space complexity is taken into account. In this paper, we present a novel randomized data structure to address the problem. The significant contributions are as follows. (1) A space-efficient counter vector sketch (CVS) are proposed, which extends the well-known bitmap sketch to sliding window settings. (2) Based on the CVS, a random update mechanism is introduced, whereby a small fixed number of entries are randomly chosen from CVS in a step and then updated. This means that the update procedure just costs constant time. (3) Furthermore, estimating cardinality by CVS just needs one-pass scan of the data. (4) Finally, a theoretical analysis is given to show the accuracy of CVS-based estimators. Our comprehensive experiments confirm that the CVS-based schema attains high accuracy, and that its time efficiency in comparison with the timestemp vector (TSV) and the auxiliary indexing method.

VATE: A trade-off between memory and preserving time for high accurate cardinality estimation under sliding time window

“…In order to improve the speed, Jingsong et al proposed CVS [22] algorithm and LRU-Sketch [23] algorithm. CVS adopts the strategy of random updating, and randomly selects a part of the counter to update each time.…”

A Super Point Detection Algorithm Under Sliding Time Windows Based on Rough and Linear Estimators

“…Therefore, we use the sliding time window to cope with these problems by adopting fine-grained moving steps [31]: the size of steps is sufficiently small so that the process can be viewed as a continuously sliding time window as suggested by its name. But detecting super points under the sliding time window is more difficult considering the intense observing frequency [32]. What's more, to work under the sliding time window, an algorithm must preserve hosts' state of a previous period [33].…”

“…Let H (x, n, A) represent a hash function that can randomly map an integer x to a number between 0 and n − 1 according to random seed parameter A [19], where n is an positive integer. For each bip in ST (t, 1), RE 0 maps ''bip'' randomly to an integer between 0 and 2 32 − 1 using the hash function H (bip, 2 32 , A 0 ) with random seed parameter A 0 . The hash function H (bip, g, A 1 ) with random seed parameter A 1 is used to map ''bip'' to an integer of 32 , A 0 ))).…”

“…For each bip in ST (t, 1), RE 0 maps ''bip'' randomly to an integer between 0 and 2 32 − 1 using the hash function H (bip, 2 32 , A 0 ) with random seed parameter A 0 . The hash function H (bip, g, A 1 ) with random seed parameter A 1 is used to map ''bip'' to an integer of 32 , A 0 ))). After scanning all the elements in a time window, RE 0 estimates the cardinality based on REI .…”