Data streaming algorithms for estimating entropy of network traffic

Abstract

Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. Estimating the entropy in a streaming model to enable such fine-grained traffic analysis has been a challenging problem. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently.We present two algorithms for randomly approximating the entropy in a time and space efficient manner, applicable for use on very high speed (greater than OC-48) links. Our first algorithm for entropy estimation, inspired by the seminal work of Alon et al. for estimating frequency moments, has strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the efficiency can be substantially enhanced by separating the high-frequency items (or elephants), from the low-frequency items (or mice). Evaluations on real-world traffic traces from different deployment scenarios demonstrate the utility of our approaches.

Measuring the complexity of the law: the United States Code

“…While entropy measures have roots in thermodynamics, the use of entropy traces back to the early and influential work of Claude Shannon (1948Shannon ( , 1951). Shannon's work has proven important to a number of fields including cryptography, machine learning and artificial intelligence (Nigam et al 1999;Lall et al 2006;Bose 2002;Ganapathi et al 2012). In addition, entropy has enjoyed appeal in other domains where information processing is relevant including physics, statistics, economics, as well as many other related disciplines (Jaynes 1957;Csiszar 1991;Golan et al 1997;Soofi 2000;Schennach 2005;Tang et al 2008).…”

Adaptive DDoS Detector Design Using Fast Entropy Computation Method

“…EBA has showed three significant benefits in [8]: increased detection sensitivity, additional diagnostic information, and useful measure between traffic groups. EBA become a popular approach against DDoS attack.…”

“…EBAs are introduced to enhance the performance of the ABA in [6,8,10]. However, the EBAs show a disadvantage in terms of time of computing entropy values.…”

High-Speed Flow Nature Identification

“…In Iustitia, we use a (δ, ǫ)-approximation algorithm to estimate S k ≡ i m ik log m ik in Formula (1). This algorithm is proposed for estimating the entropy of data streams by Lall et al in [12]. The basic idea is to use frequency moments estimation techniques [3].…”