Sketch Guided Sampling - Using On-Line Estimates of Flow Size for Adaptive Data Collection

Abstract

Abstract-Monitoring the traffic in high-speed networks is a data intensive problem. Uniform packet sampling is the most popular technique for reducing the amount of data the network monitoring hardware/software has to process. However, uniform sampling captures far less information than can be potentially obtained with the same overall sampling rate. This is because uniform sampling (unnecessarily) draws the vast majority of samples from large flows, and very few from small and medium flows. This information loss on small and medium flows significantly affects the accuracy of the estimation of various network statistics.In this work, we develop a new packet sampling methodology called "sketch-guided sampling" (SGS), which offers better statistics than obtainable from uniform sampling, given the same number of raw samples gathered. Its main idea is to make the probability with which an incoming packet is sampled a decreasing sampling function ¢ of the size of the flow the packet belongs to. This way our scheme is able to significantly increase the packet sampling rate of the small and medium flows at slight expense of the large flows, resulting in much more accurate estimations of various network statistics. However, the exact sizes of all flows are available only if we keep per-flow information for every flow, which is prohibitively expensive for highspeed links. Our SGS scheme solves this problem by using a small (lossy) synopsis data structure called counting sketch to encode the approximate sizes of all flows. Our evaluation on real-world Internet traffic traces shows that our sampling theory based the approximate flow size estimates from the counting sketch works almost as well as if we know the exact sizes of the flows.

Uncovering Global Icebergs in Distributed Streams: Results and Implications

“…This can be answered by studying the estimation error for reverting sampling. With sampling rate p, an item of count c has estimation variance c(1 -p)/p [23]. In comparison, the sketches that we are using have considerably less error.…”

Modeling Residual-Geometric Flow Sampling

“…Instead of using one uniform probability for all flows as in [10], [25], another direction in packet sampling is to compute p i (c) for each flow i based on its currently observed size c. This approach has been studied by two independent papers, Sketch-Guided Sampling (SGS) [20] and Adaptive Non-Linear Sampling (ANLS) [15]. A common feature of these two methods is to sample a new flow with probability 1 and then monotonically decrease p i (c) as c grows.…”

“…The main problem in this line of work is to scale measurement algorithms to achieve certain objectives (e.g., accuracy) while satisfying real-time resource constraints (e.g., fixed memory consumption and per-packet processing delay) of high-speed Internet routers. This is commonly accomplished (e.g., [5], [6], [7], [8], [9], [10], [11], [14], [15], [17], [21], [18], [19], [20], [22], [25], [30]) by reducing the amount of information a router has to store in its internal tables, which comes at the expense of deploying special estimation techniques that can recover metrics of interest from the collected samples.…”

“…In this paper, we study two problems in the general area of measuring flow sizes -1) determining the number of packets transmitted by "elephant" flows [11], [15], [17], [21], [20], [22] and 2) building the distribution of flow sizes seen by the router in some time window [7], [18], [30] -coupled in a single measurement technique. The former problem arises in usage-based accounting and traffic engineering [6], [11], [12], [13], [26], while the latter has many security applications such as anomaly and intrusion detection [1], [23], [16].…”

Boosting the scalability of botnet detection using adaptive traffic sampling

“…Recently researchers proposed approaches to focus on sampling packets in small flows. Kumar et al [1] and Hu et al [9] proposed algorithms to sample packets in small flows. However, their overall sampling rate depends on the Zipfian nature [19] of Internet and thus they cannot achieve a pre-defined target sampling rate.…”