The flow size distribution is a useful metric for traffic modeling and management. It is well known however that its estimation based on sampled data is problematic. Previous work has shown that flow sampling (FS) offers enormous statistical benefits over packet sampling, however it suffers from high resource requirements and is not currently used in routers. In this paper we present Dual Sampling, which can to a large extent provide flow-sampling-like statistical performance for packet-sampling-like computational cost. Our work is grounded in a Fisher information based approach recently used to evaluate a number of sampling schemes, excluding however FS, for TCP flows. We show how to revise and extend the approach to include FS as well as DS and others, and how to make rigorous and fair comparisons. We show how DS significantly outperforms other packet based methods, but also prove that DS is inferior to flow sampling. However, since DS is a two-parameter family of methods which includes FS as a special case, DS can be used to approach flow sampling continuously. We then describe a packet sampling based implementation of DS and analyze its key computational costs to show that router implementation is feasible. Our approach offers insights into many issues, including how the notions of 'flow quality' and 'packet gain' can be used to understand the relative performance of methods, and how the problem of optimal sampling can be formulated. Our work is theoretical with some simulation support and a case study on Internet data.

No abstract

No abstract

The flow size distribution is a useful metric for traffic modeling and management. Its estimation based on sampled data, however, is problematic. Previous work has shown that flow sampling (FS) offers enormous statistical benefits over packet sampling but high resource requirements precludes its use in routers. We present Dual Sampling (DS), a two-parameter family, which, to a large extent, provide FS-like statistical performance by approaching FS continuously, with just packet-samplinglike computational cost. Our work utilizes a Fisher information based approach recently used to evaluate a number of sampling schemes, excluding FS, for TCP flows. We revise and extend the approach to make rigorous and fair comparisons between FS, DS and others. We show how DS significantly outperforms other packet based methods, including Sample and Hold, the closest packet sampling-based competitor to FS. We describe a packet sampling-based implementation of DS and analyze its key computational costs to show that router implementation is feasible. Our approach offers insights into numerous issues, including the notion of 'flow quality' for understanding the relative performance of methods, and how and when employing sequence numbers is beneficial. Our work is theoretical with some simulation support and case studies on Internet data. Index Terms-Fisher information, flow size distribution, Internet measurement, router measurement, sampling.

No abstract

Traffic matrices are used in many network engineering tasks, for instance optimal network design. Unfortunately, measurements of these matrices are error-prone, a problem that is exacerbated when they are extrapolated to provide the predictions used in planning. Practical network design and management should consider sensitivity to such errors, but although robust optimisation techniques exist, it seems they are rarely used, at least in part because of the difficulty in generating an ensemble of admissible traffic matrices with a controllable error level. We address this problem in our paper by presenting a fast and flexible technique of generating synthetic traffic matrices. We demonstrate the utility of the method by presenting a methodology for robust network design based on adaptation of the mean-risk analysis concept from finance.

We introduce a new method of data collection for flow size estimation, the optimized flow sampling sketch, which combines the optimal properties of flow sampling with the computational advantages of a counter array sketch. Using Fisher information as a definitive basis of comparison, we show that the statistical efficiency of the method is within a constant factor of that of flow sampling, which is known to be optimal but which cannot be implemented without a flow table, which has higher memory and computational costs. In the process, we derive new results on the Fisher information theoretic and variance properties of the counter array sketch, proving that an overloaded sketch actually destroys information. We revisit the 'eviction sketch' of Ribeiro et al. using the Fisher information framework. We show that its performance is much higher than previously supposed, and we define a new method, the optimized eviction sketch, which has very high efficiency. We compare these methods against each other and a third skampling method, sketch guided sampling, theoretically, on models and on data.Index Terms-Cramér-Rao lower bounds, internet, maximum likelihood estimation, sampling methods, sketching.

No abstract