Mining anomalies using traffic feature distributions

Abstract

The increasing practicality of large-scale flow capture makes it possible to conceive of traffic analysis methods that detect and identify a large and diverse set of anomalies. However the challenge of effectively analyzing this massive data source for anomaly diagnosis is as yet unmet. We argue that the distributions of packet features (IP addresses and ports) observed in flow traces reveals both the presence and the structure of a wide range of anomalies. Using entropy as a summarization tool, we show that the analysis of feature distributions leads to significant advances on two fronts: (1) it enables highly sensitive detection of a wide range of anomalies, augmenting detections by volume-based methods, and (2) it enables automatic classification of anomalies via unsupervised learning. We show that using feature distributions, anomalies naturally fall into distinct and meaningful clusters. These clusters can be used to automatically classify anomalies and to uncover new anomaly types. We validate our claims on data from two backbone networks (Abilene and Geant) and conclude that feature distributions show promise as a key element of a fairly general network anomaly diagnosis framework.

Resource-aware placement of softwarised security services in cloud data centers

“…Network-Wide class: Represent modules detecting network-wide threats such as probes and worms which cause distribution changes in traffic features that can be observed at high aggregation levels [10]. Therefore, this class monitors aggregate packets/flows and consequently they can be allocated to centralised/core locations where aggregate traffic to multiple destinations can be processed.…”

Large-scale IP network behavior anomaly detection and identification using substructure-based approach and multivariate time series mining

“…There is considerable interest in using entropy-based analysis of traffic feature distributions for anomaly detection [49,50]. Entropy-based metrics are tempting since they provide more fine-grained insights into traffic structure than traditional traffic volume analysis.While previous work has illuminated the benefits of using the entropy of different traffic distributions in isolation to detect anomalies, there has been little effort in comprehensively understanding the detection power provided by entropy-based analysis of multiple traffic distribution used in connection with each other.…”

An Entropy-based Method for Attack Detection in Large Scale Network

“…Lakhina et al argue that the distributions of packet features (IP addresses and ports) observed in flow traces reveal both the presence and structure of a wide range of anomalies. Using entropy as a summarization tool to analyze traffic from two backbone networks, they found that it enables highly sensitive detection of a wide range of anomalies, augmenting detections by volume-based methods [8]. Brauckhoff ind that entropy-based summarizations of packet and flow counts are affected less by sampling than volume-based method in large networks [9].…”