Fine-grained probability counting for cardinality estimation of data streams

Wang, Lun; Yang, Tong; Wang, Hao; Jiang, Jie; Cai, Zekun; Cui, Bin; Li, Xiaoming

doi:10.1007/s11280-018-0583-0

Cited by 9 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Filter [58] Virtual Register Sharing [67] Virtual HyperLogLog [70] Counter tree [75] Braids [22] SketchVisor [135] MAPLE [160] NitroSketch [55], R HHH [54] CASE [174] CASE [183], RCC [87] HeavyGuardian [100] Sliding HLL [127] Fast sketch [120], OPA [115] Sequential hashing [112] Modular hashing [119] Bitcount [143], CBF [50] SketchLearn [20] Group Testing [159] HeavyKeeper [99] HeavyGuardian [100] ICE-Buckets [52] HyperSight [153] MultiResBitmap [170] HLL-TailCut+ [74], RAP [93], Frequent [94] ACE [125], Conservative update [88] CountMax [96], LCF [184], LR(T) [185], SpaceSaving [95], CSketch [89], CSM [90] CHHFR [171], Refined LL [166] CounterMap [198], IMP [118] SketchVisor [135], WCSS[128] SWAMP [129], RCD [113] Defeat …”

Section: Improve Accuracymentioning

confidence: 99%

“…Refined LogLog [171] uses a fine-grained common ratio instead of 2 used in FM Sketch [66] and LogLog [67] to narrow the gaps between two recordable values. By using geometrically hash functions with smaller common ratio, Refined LogLog breaks the restrict that the former method can only record 2 k , which may have a large gap with the accurate number.…”

Section: ) Time Dimensionmentioning

confidence: 99%

“…MinCount [170] 1.00/ √ m 32-bit keys PCSA [66] 0.78/ √ m 32-bit register LogLog [67] 1.30/ √ m 5-bit register HLL [68] 1.04/ √ m 5-bit register Sliding HLL [70] 1.04/ √ m 5-bit register HLL-TailCut+ [69] 1.00/ √ m 3-bit register Refined LL [171] 1.00/ √ m 5-bit register Cohen and Nezti [157] −− −− per-flow measurement. Apart from the CM sketch [27] and its variants, most research focus on the skewness characteristic of traffic stream.…”

Section: ) Time Dimensionmentioning

confidence: 99%

See 2 more Smart Citations

A survey of sketches in traffic measurement: Design, Optimization, Application and Implementation

Li¹,

Luo²,

Guo³

et al. 2020

Preprint

View full text Add to dashboard Cite

Network measurement probes the underlying network to support upper-level decisions such as network management, network update, network maintenance, network defense and beyond. Due to the massive, speedy, unpredictable features of network flows, sketches are widely implemented in measurement nodes to record the frequency or estimate the cardinality of flows approximately. At their cores, sketches usually maintain one or multiple counter array(s), and relies on hash functions to select the counter(s) for each flow. Then the space-efficient sketches from the distributed measurement nodes are aggregated to provide statistics of the undergoing flows. Currently, tremendous redesigns and optimizations have been proposed to further improve the sketches for better network measurement performance. However, the existing reviews or surveys mainly focus on one particular aspect of measurement tasks. Researchers and engineers in the network measurement community desire an all-in-one survey which covers the whole processing pipeline of sketch-based network measurement. To this end, we present the first comprehensive survey in this area. We first introduce the preparation of flows for measurement, then detail the most recent investigations of design, aggregation, decoding, application and implementation of sketches for network measurement. To summary the existing efforts, we conduct an in-depth study of the existing literature, covering more than 80 sketch designs and optimization strategies. Furthermore, we conduct a comprehensive analysis and qualitative/quantitative comparison of the sketch designs. Finally, we highlight the open issues for future sketch-based network measurement research.

show abstract

Section: Improve Accuracymentioning

confidence: 99%

Section: ) Time Dimensionmentioning

confidence: 99%

Section: ) Time Dimensionmentioning

confidence: 99%

See 1 more Smart Citation

A survey of sketches in traffic measurement: Design, Optimization, Application and Implementation

Li¹,

Luo²,

Guo³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…e method can measure the cardinality of each host accurately, but it is not practical to process massive network traffic due to large memory overhead [27,28]. Sampling-based methods [29,30] and data stream-based methods [31][32][33] are proposed to tackle the challenges caused by big network traffic. However, the estimation accuracy of sampling-based methods depends on the sampling rate, that is, they maintain a small number of distinct connections at the small sampling rate, but the accuracy of cardinality estimation decreases; on the contrary, the accuracy of cardinality estimation increases at the high sampling rate, but the memory overhead also increases.…”

Section: Introductionmentioning

confidence: 99%

An Efficient Method for Detecting Supernodes Using Reversible Summary Data Structures in the Distributed Monitoring Systems

Zhou

Jin

2022

Security and Communication Networks

View full text Add to dashboard Cite

Supernode detection has many applications in detecting network attacks, assisting resource allocation, etc. As 5G/IoT networks constantly grow, big network traffic brings a great challenge to collect massive traffic data in compact and real-time way. Previous works focus on detecting supernodes in a measurement point, while only a few works consider it in the distributed monitoring system. Moreover, they are not able to measure two types of node cardinalities simultaneously and reconstruct labels of supernodes efficiently due to large calculation and memory cost. To address these problems, we propose a novel reversible and distributed traffic summarization called RDS to simultaneously measure source and destination cardinalities for detecting supernodes in the distributed monitoring system. The basic idea of our approach is that each monitor generates a summary data structure using the coming packets and sends the summary data structure to the controller; then, the controller aggregates the received summary data structures, estimates node cardinalities, and reconstructs labels of supernodes according to the aggregated summary data structure. The experimental results based on real network traffic demonstrate that the proposed approach can detect up to 96% supernodes with a low memory requirement in comparison with state-of-the-art approaches.

show abstract