Fisher Information in Flow Size Distribution Estimation

Abstract

The flow size distribution is a useful metric for traffic modeling and management. Its estimation based on sampled data, however, is problematic. Previous work has shown that flow sampling (FS) offers enormous statistical benefits over packet sampling but high resource requirements precludes its use in routers. We present Dual Sampling (DS), a two-parameter family, which, to a large extent, provide FS-like statistical performance by approaching FS continuously, with just packet-samplinglike computational cost. Our work utilizes a Fisher information based approach recently used to evaluate a number of sampling schemes, excluding FS, for TCP flows. We revise and extend the approach to make rigorous and fair comparisons between FS, DS and others. We show how DS significantly outperforms other packet based methods, including Sample and Hold, the closest packet sampling-based competitor to FS. We describe a packet sampling-based implementation of DS and analyze its key computational costs to show that router implementation is feasible. Our approach offers insights into numerous issues, including the notion of 'flow quality' for understanding the relative performance of methods, and how and when employing sequence numbers is beneficial. Our work is theoretical with some simulation support and case studies on Internet data. Index Terms-Fisher information, flow size distribution, Internet measurement, router measurement, sampling.

Optimal Skampling for the Flow Size Distribution

“…We define the two flow size summary methods: flow sampling (FS) and the counter sketch (Sk), that are central to the study of both FSS and ESk, and derive their Fisher Information. In the case of FS we recall the results from [20]. For Sk we follow the approach from [21] but offer a more complete (and corrected) treatment of the basic setup.…”

“…As discussed in detail in [20], we can assume N f is known. Of these flows, M k have size k packets, 1 ≤ k ≤ W , where W < ∞ is the maximum flow size.…”

“…In our prior work [8], [19], [20], we have shown that FS is statistically superior to all currently proposed alternatives, including Sample and Hold [5], and methods such as Dual Sampling [20] and others [17] exploiting TCP sequence number information. 1 This conclusion was based on a rigorous evaluation, in terms of Fisher Information, of the inherent abilities of the sampling methods as information collectors.…”

“…The OFSS is an FSS tuned according to an optimal tradeoff between the information gathering strengths and weaknesses of the FS and Sk components. 1 Caveats exist in special cases of no practical importance [20]. 0018-9448 © 2015 IEEE.…”

Challenges of Forecasting and Measuring a Complex Networked World

“…We discuss the theoretic foundations of this paradox and its connections with known information theoretic measures such as Shannon capacity. We also discuss the implications of this paradox on the scalability of big data applications and show how information theory tools -such as Fisher information [3,8] -can be used to design more accurate and scalable methods for network analytics [6,8,10]. The second part of the talk focuses on how these results impact our ability to perform network analytics when network data is only available through crawlers and the complete network topology is unknown [1,4,5,9].…”