Sketching and Streaming Entropy via Approximation Theory

Abstract

We conclude a sequence of work by giving near-optimal sketching and streaming algorithms for estimating Shannon entropy in the most general streaming model, with arbitrary insertions and deletions. This improves on prior results that obtain suboptimal space bounds in the general model, and near-optimal bounds in the insertion-only model without sketching. Our high-level approach is simple: we give algorithms to estimate Renyi and Tsallis entropy, and use them to extrapolate an estimate of Shannon entropy. The accuracy of our estimates is proven using approximation theory arguments and extremal properties of Chebyshev polynomials, a technique which may be useful for other problems. Our work also yields the best-known and near-optimal additive approximations for entropy, and hence also for conditional entropy and mutual information.

Distributed monitoring of conditional entropy for anomaly detection in streams

“…While possessing the significant advantage of needing few modeling assumptions about what constitutes normal and abnormal traffic, entropy-based methods incur substantial computational cost, which presented an obstacle to their practical adoption. Recent advances in streaming algorithms [3], [6], [12], [16] pave the way to overcoming this obstacle. Reducing the computation cost opens the way not only for singlevariable entropy-based anomaly detection, but also for analysis of pairwise feature dependencies, which can be an efficient anomaly detection tool.…”

“…The algorithms of Bhuvanagiri and Ganguly [3] and Harvey et al [12] are in a class of algorithms called sketching algorithms.…”

“…Space requirement Naïve O(m log n) Lall et al 3 [16] O((1/ε) 2 log(1/δ) log m log n) Bhuvanagiri & Ganguly [3] O((1/ε 3 ) log(1/δ) log 5 m) Harvey et al [12] O((1/ε) 2 log(1/δ) log m log n log(mn)) available (see Table I) for estimating entropy, the case for conditional entropy turns out to be different. Indyk and McGregor [13] showed that estimating conditional entropy with ε-multiplicative error requires Ω(m) space, so that no non-trivial savings are possible over naïve computation even with randomized, approximation algorithms.…”

“…Thus, one has to consider randomized approximation algorithms to have any hope of computing entropy efficiently. This problem of monitoring entropy efficiently has received widespread interest in the theory community, and several data streaming algorithms are now known [2], [3], [6], [7], [11], [12], [16] that utilize significantly lower memory than straightforward computation. However, even such algorithms fail to keep up with everincreasing channel bandwidths observed at a typical NOC of a large organization, especially considering that monitoring of several stream features is desirable for accurate detection.…”

Space-Efficient Estimation of Statistics Over Sub-Sampled Streams

“…Therefore, by using an existing entropy estimation algorithm (e.g., [25]) to multiplicatively estimate H(g) we have a constant factor approximation to H(f) if H(f) = ω(p −1/2 n −1/6 ). The next theorem follows directly from Proposition 1 and Lemma 10.…”