Extracting hidden anomalies using sketch and non Gaussian multiresolution statistical detection procedures

Abstract

A new profile-based anomaly detection and characterization procedure is proposed. It aims at performing prompt and accurate detection of both short-lived and long-lasting low-intensity anomalies, without the recourse of any prior knowledge of the targetted traffic. Key features of the algorithm lie in the joint use of random projection techniques (sketches) and of a multiresolution non Gaussian marginal distribution modeling. The former enables both a reduction in the dimensionality of the data and the measurement of the reference (i.e., normal) traffic behavior, while the latter extracts anomalies at different aggregation levels. This procedure is used to blindly analyze a large-scale packet trace database collected on a trans-Pacific transit link from 2001 to 2006. It can detect and identify a large number of known and unknown anomalies and attacks, whose intensities are low (down to below one percent). Using sketches also makes possible a real-time identification of the source or destination IP addresses associated to the detected anomaly and hence their mitigation.

Workload Modeling for Computer Systems Performance Evaluation

“…Another technique that has been proposed for removing anomalies from Internet traffic data (or, rather, for negating their effect on evaluation results) is the following [176]. First, hash the data into a number of groups based on some select workload attribute, for example a flow's source IP address or destination port.…”

UNADA: Unsupervised Network Anomaly Detection Using Sub-space Outliers Ranking

“…of packets, bytes, or new flows) and/or particular traffic features (e.g., distribution of IP addresses and ports), using either single-link measurements or network-wide data. A non-exhaustive list of standard methods includes the use of signal processing techniques (e.g., ARIMA modeling, wavelets-based filtering) on single-link traffic measurements [1], Kalman filters [4] for network-wide anomaly detection, and Sketches applied to IP-flows [5,6].…”

“…The traces we shall work with consist of traffic from one of the trans-pacific links between Japan and the U.S.. MAWI traces are not labeled, but some previous work on anomaly detection has been done on them [6,15]. In particular, [15] detects network attacks using a signature-based approach, while [6] detects both attacks and anomalous flows using non-Gaussian modeling. We shall therefore refer to the combination of results obtained in both works as our ground truth for MAWI traffic.…”

Uncovering Relations between Traffic Classifiers and Anomaly Detectors via Graph Theory

“…-hash based (sketch) anomaly detectors [11,12] usually report only IP addresses and corresponding time bin, no other information (e.g. port number) describes identified anomalies.…”

“…One consists of random projection techniques (sketches) and multi-resolution gamma modeling [11]. Hereafter we call it as the gamma-based method.…”