Discount Counting for Fast Flow Statistics on Flow Size and Flow Volume

Hu, Chengchen; Liu, Bin; Zhao, Hongbo; Chen, Kai; Chen, Yan; Cheng, Yu; Wu, Hao

doi:10.1109/tnet.2013.2270439

Cited by 33 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There are two categories of methods widely applied in super host identification: sampling-based and sketch-based. Some studies have pointed out that sampling-based methods exhibit poor accuracy and high memory consumption or storage cost [10], [31]. Sketch-based methods have been attracting much attention in recent years due to their special advantages over the sampling-based methods.…”

Section: Related Workmentioning

confidence: 99%

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Jing

Yan

Han

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

Super host refers to the host that has a high cardinality or exhibits a big change in a network. Facing big-volume network traffic, sketches have been widely applied to identify super hosts in an efficient and accurate way. However, most sketches cannot flexibly balance memory usage and accuracy in host cardinality estimation. Setting an inappropriate counter size for a sketch could either lead to inaccurate host cardinality estimation or cause memory waste. In order to solve this issue, we propose a novel extensible and reversible sketch, named ExtendedSketch, to achieve accurate super host identification with high memory efficiency. The core idea of ExtendedSketch is to monitor low-cardinality hosts with small-sized counters while dynamically extending the size of counters when monitoring high-cardinality hosts by applying an adaptive extension strategy. Such the strategy can adaptively increase counter size according to network traffic status at runtime, which not only ensures the accuracy of high-cardinality host estimation but also avoids unnecessary memory consumption. We perform theoretical analysis and conduct a series of experimental evaluations on ExtendedSketch based on real world network traffic. Experimental results show that under same memory usage, compared to the state-of-the-art, ExtendedSketch achieves 1.4 ∼ 7.5 times smaller error rate in estimating host cardinality with 1.9 ∼ 26.7 times better accuracy on super host identification and 95 ∼ 2 15 times faster speed on abnormal address reconstruction. Its advance in accuracy and efficiency demonstrates the practical significance of ExtendedSketch for super host identification.

show abstract

Section: Related Workmentioning

confidence: 99%

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Jing

Yan

Han

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

show abstract

“…ey also propose a nonuniform packet sampling algorithm based on sketch counting [12]. Hu et al [10,14] propose another nonuniform packet sampling algorithm, called adaptive nonlinear sampling (ANLS), for estimating flow lengths per each flow and then adopt this method to flow volume [11]. In our experiments, we are going to use ANLS as an example of flow-based sampling methods since it is the current state-of-the-art nonuniform sampling method.…”

Section: Related Workmentioning

confidence: 99%

“…Flow-based adaptive sampling methods [9][10][11][12][13][14] were proposed for more accurate flow length estimation. In these methods, each incoming packet is processed and then sampled with a probability that is a function of the current sampled length of the flow that the packet belongs to.…”

Section: Introductionmentioning

confidence: 99%

Estimating Network Flow Length Distributions via Bayesian Nonnegative Tensor Factorization

Kurt

Cemgil

Kurt

et al. 2019

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

In this paper, we develop a framework to estimate network flow length distributions in terms of the number of packets. We model the network flow length data as a three-way array with day-of-week, hour-of-day, and flow length as entities where we observe a count. In a high-speed network, only a sampled version of such an array can be observed and reconstructing the true flow statistics from fewer observations becomes a computational problem. We formulate the sampling process as matrix multiplication so that any sampling method can be used in our framework as long as its sampling probabilities are written in matrix form. We demonstrate our framework on a high-volume real-world data set collected from a mobile network provider with a random packet sampling and a flow-based packet sampling methods. We show that modeling the network data as a tensor improves estimations of the true flow length histogram in both sampling methods.

show abstract

“…Note that the relative error is independent of n, and therefore max = . For comparison, DISCO's [29] estimation function,…”

Section: Rm Sre [N] = ∀Nmentioning

confidence: 99%

ICE Buckets: Improved Counter Estimation for Network Measurement

Einziger

Fellman

Friedman

et al. 2018

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

Measurement capabilities are essential for a variety of network applications, such as load balancing, routing, fairness and intrusion detection. These capabilities require large counter arrays in order to monitor the traffic of all network flows. While commodity SRAM memories are capable of operating at line speed, they are too small to accommodate large counter arrays. Previous works suggested estimators, which trade precision for reduced space. However, in order to accurately estimate the largest counter, these methods compromise the accuracy of the smaller counters. In this work, we present a closed form representation of the optimal estimation function. We then introduce Independent Counter Estimation Buckets (ICE-Buckets), a novel algorithm that improves estimation accuracy for all counters. This is achieved by separating the flows to buckets and configuring the optimal estimation function according to each bucket's counter scale. We prove a tighter upper bound on the relative error and demonstrate an accuracy improvement of up to 57 times on real Internet packet traces.

show abstract

Discount Counting for Fast Flow Statistics on Flow Size and Flow Volume

Cited by 33 publications

References 25 publications

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Estimating Network Flow Length Distributions via Bayesian Nonnegative Tensor Factorization

ICE Buckets: Improved Counter Estimation for Network Measurement

Contact Info

Product

Resources

About