Data streaming algorithms for estimating entropy of network traffic

Abstract

Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. Estimating the entropy in a streaming model to enable such fine-grained traffic analysis has been a challenging problem. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently.We present two algorithms for randomly approximating the entropy in a time and space efficient manner, applicable for use on very high speed (greater than OC-48) links. Our first algorithm for entropy estimation, inspired by the seminal work of Alon et al. for estimating frequency moments, has strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the efficiency can be substantially enhanced by separating the high-frequency items (or elephants), from the low-frequency items (or mice). Evaluations on real-world traffic traces from different deployment scenarios demonstrate the utility of our approaches.

Distributed monitoring of conditional entropy for anomaly detection in streams

“…Space requirement Naïve O(m log n) Lall et al 3 [16] O((1/ε) 2 log(1/δ) log m log n) Bhuvanagiri & Ganguly [3] O((1/ε 3 ) log(1/δ) log 5 m) Harvey et al [12] O((1/ε) 2 log(1/δ) log m log n log(mn)) available (see Table I) for estimating entropy, the case for conditional entropy turns out to be different. Indyk and McGregor [13] showed that estimating conditional entropy with ε-multiplicative error requires Ω(m) space, so that no non-trivial savings are possible over naïve computation even with randomized, approximation algorithms.…”

“…We now detail the performance in conditional entropy estimation of the two algorithms we consider: the HSS algorithm and Algorithm 1 of Lall et al [16]. Our tests in this section were performed on Trace 1 and follow along the lines of the evaluations in [16].…”

“…The information-theoretic statistic of empirical entropy (or simply entropy) has received a lot of attention in this respect [4], [10], [16], [17], [18], [19], [21], [22]. Computing entropy in the straightforward manner, by maintaining counters to keep track of the distribution histogram, is expensive memory-wise and computationally.…”

“…Computing entropy in the straightforward manner, by maintaining counters to keep track of the distribution histogram, is expensive memory-wise and computationally. Lall et al [16] showed that computing entropy either deterministically or precisely requires linear space, therefore in these cases no non-trivial savings can be achieved over naïve computation. Thus, one has to consider randomized approximation algorithms to have any hope of computing entropy efficiently.…”

“…Thus, one has to consider randomized approximation algorithms to have any hope of computing entropy efficiently. This problem of monitoring entropy efficiently has received widespread interest in the theory community, and several data streaming algorithms are now known [2], [3], [6], [7], [11], [12], [16] that utilize significantly lower memory than straightforward computation. However, even such algorithms fail to keep up with everincreasing channel bandwidths observed at a typical NOC of a large organization, especially considering that monitoring of several stream features is desirable for accurate detection.…”