A data streaming algorithm for estimating entropies of od flows

Abstract

Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.

Sketching and Streaming Entropy via Approximation Theory

“…Many algorithmic problems in this model are now well-understood, for example, the problem of estimating frequency moments [1,2,10,18,32,35]. More recently, several researchers have studied the problem of estimating the empirical entropy of a stream [3,6,7,12,13,37].…”

Towards a Universal Sketch for Origin-Destination Network Measurements

“…Such limitations have constrained many network management tasks, and forced significant research and engineering efforts to be expended on finding creative solutions. In particular, targeting the aforementioned inadequacy of counting/accounting capabilities of existing routers, many data streaming and sketching techniques (e.g., [14,15,33]) have been proposed to estimate the important statistics of, and to approximately answer various queries concerning, traffic going through a network link.…”

Streaming algorithms for estimating entropy

“…Estimating empirical entropy in the streaming model is a problem that has been studied in the recent literature [3], [5], [6], [9], [18]. Two sorts of estimates are typically of interest: multiplicative (1+ ) approximations, and additive approximations.…”

“…In contrast, the algorithm of Bhuvanagiri and Ganguly [3] can handle deletions, but the space required is roughly O( −3 log 7 m) words. Zhao et al [18] give practical methods for estimating the so-called entropy norm of a stream, defined to be n i=1 A i log A i , where A is not normalized to be a distribution.…”

“…We remark that the algorithm of Zhao et al [18] also uses frequency moment estimation, although their algorithm is intended for estimating the entropy norm, and seems to only work for certain ranges of parameters.…”