Optimal Skampling for the Flow Size Distribution

Abstract

We introduce a new method of data collection for flow size estimation, the optimized flow sampling sketch, which combines the optimal properties of flow sampling with the computational advantages of a counter array sketch. Using Fisher information as a definitive basis of comparison, we show that the statistical efficiency of the method is within a constant factor of that of flow sampling, which is known to be optimal but which cannot be implemented without a flow table, which has higher memory and computational costs. In the process, we derive new results on the Fisher information theoretic and variance properties of the counter array sketch, proving that an overloaded sketch actually destroys information. We revisit the 'eviction sketch' of Ribeiro et al. using the Fisher information framework. We show that its performance is much higher than previously supposed, and we define a new method, the optimized eviction sketch, which has very high efficiency. We compare these methods against each other and a third skampling method, sketch guided sampling, theoretically, on models and on data.Index Terms-Cramér-Rao lower bounds, internet, maximum likelihood estimation, sampling methods, sketching.

Skampling for the Flow Duration Distribution

“…Second, as a result, the considered inference procedures show superior performance at suitable sampling rates, which can be tuned at the sampling stage. These ideas have been suggested explicitly and studied in [2] for the distribution of flow sizes.…”

“…The performance of the estimator depends not only on the load factor ρ q but also on the number of sampled busy periods in the estimation. For a fixed p, the mean busy cycle is (e λqE[D] − 1)/λ q + 1/λ q by using (2). The minimum value of the mean busy cycle is attained when λ q = 1/E[D], yielding ρ q = 1 and p/A = 1/(λE[D]).…”

“…Our work is inspired by a combined sampling and sketching approach for the flow size distribution, called skampling (a portmanteau of sketching and sampling), developed in [2]. The approach in [2] comprises three main components. First, sampling is performed at the flow level with probability p (all packets of a flow, or none, will pass).…”

“…Because duration measurement involves timekeeping, the situation is naturally more complex than the case of flow size [2]. Whereas a single counter counting arrivals over a measurement interval was sufficient in that context, here an entry in the sketch array may involve multiple counters, and require state dependent actions rather than a simple increment.…”

“…Unless stated otherwise, we focus on a single sketch entry, fed by a stream of flows which we think of as associated to an M/G/∞ queue of arrival intensity λ q . We follow the approach of [2] by considering sketch entries to be mutually independent. Final sketch estimates can therefore be obtained by combining those from each entry in a straightforward way, effectively increasing sample size by a factor of A.…”