Accurate and Efficient Traffic Monitoring Using Adaptive Non-Linear Sampling Method

Abstract

Sampling technology has been widely deployed in measurement systems to control memory consumption and processing overhead. However, most of the existing sampling methods suffer from large estimation errors in analyzing small-size flows. To address the problem, we propose a novel adaptive non-linear sampling (ANLS) method for passive measurement. Instead of statically configuring the sampling rate, ANLS dynamically adjusts the sampling rate for a flow depending on the number of packets having been counted. We provide the generic principles guiding the selection of sampling function for sampling rate adjustment. Moreover, we derive the unbiased flow size estimation, the bound of the relative error, and the bound of required counter size for ANLS. The performance of ANLS is thoroughly studied through theoretic analysis and experiments under synthetic/real network data traces, with comparison to several related sampling methods. The results demonstrate that the proposed ANLS can significantly improve the estimation accuracy, particularly for small-size flows, while maintain a memory and processing overhead comparable to existing methods.

Boosting the scalability of botnet detection using adaptive traffic sampling

“…Recently researchers proposed approaches to focus on sampling packets in small flows. Kumar et al [1] and Hu et al [9] proposed algorithms to sample packets in small flows. However, their overall sampling rate depends on the Zipfian nature [19] of Internet and thus they cannot achieve a pre-defined target sampling rate.…”

Modeling residual-geometric flow sampling

“…Instead of using one uniform probability for all flows as in [10], [25], another direction in packet sampling is to compute p i (c) for each flow i based on its currently observed size c. This approach has been studied by two independent papers, Sketch-Guided Sampling (SGS) [20] and Adaptive Non-Linear Sampling (ANLS) [15]. A common feature of these two methods is to sample a new flow with probability 1 and then monotonically decrease p i (c) as c grows.…”

“…The main problem in this line of work is to scale measurement algorithms to achieve certain objectives (e.g., accuracy) while satisfying real-time resource constraints (e.g., fixed memory consumption and per-packet processing delay) of high-speed Internet routers. This is commonly accomplished (e.g., [5], [6], [7], [8], [9], [10], [11], [14], [15], [17], [21], [18], [19], [20], [22], [25], [30]) by reducing the amount of information a router has to store in its internal tables, which comes at the expense of deploying special estimation techniques that can recover metrics of interest from the collected samples.…”

“…In this paper, we study two problems in the general area of measuring flow sizes -1) determining the number of packets transmitted by "elephant" flows [11], [15], [17], [21], [20], [22] and 2) building the distribution of flow sizes seen by the router in some time window [7], [18], [30] -coupled in a single measurement technique. The former problem arises in usage-based accounting and traffic engineering [6], [11], [12], [13], [26], while the latter has many security applications such as anomaly and intrusion detection [1], [23], [16].…”

Discount Counting for Fast Flow Statistics on Flow Size and Flow Volume

“…However, without any compressing on the statistics, this method still faces the risk to overflow their counters when counting flow bytes or needs a quite large fast DRAM. Previous SRAM-based solution susually employed random sampling as a common approach to control the memory consumption of flow size statistics [2], [6], [7], [9]. However, simple extensions of sampling methods for flow volume counting will lead to awkward performance in accuracy or processing speed.…”