A scalable, efficient and informative approach for anomaly‐based intrusion detection systems: theory and practice

Abstract

SUMMARYIn this paper, we present the design and implementation of a new approach for anomaly detection and classification over high speed networks. The proposed approach is based first of all on a data reduction phase through flow sampling by focusing mainly on short lived flows. The second step is then a random aggregation of some descriptors such as a number of SYN packets per flow in two different data structures called Count Min Sketch and Multi-Layer Reversible Sketch. A sequential change point detection algorithm continuously monitors the sketch cell values. An alarm is raised if a significant change is identified in cell values. With an appropriate definition of the combination of IP header fields that should be used to identify one flow, we are able not only to detect the anomaly but also to classify the anomaly as DoS, DDoS or flash crowd, network scanning and port scanning. We validate our framework for anomaly detection on various real world traffic traces and demonstrate the accuracy of our approach on these real-life case studies. Our analysis results from online implementation of our algorithm over measurements gathered by a DAG sniffing card are very attractive in terms of accuracy and response time. The proposed approach is very effective in detecting and classifying anomalies, and in providing information by extracting the culprit flows with a high level of accuracy. Copyright

Sketch-based multidimensional IDS: A new approach for network anomaly detection

“…In general they act on monodimensional time series and, to consider different traffic features, the same basic algorithm is sequentially repeated [2], [3], [1]. The above-mentioned papers represent the starting point for the present work, which extended the structure of the IDS described in [1] to operate with vectorial sequences.…”

“…Sketches can not be considered as a detection method, nevertheless they can be used as a building block of several AD systems [12], [13], [15], [14], [16], [17], [18], [19], [2], [1]. Indeed, the use of sketches corresponds to a random aggregation that "efficiently" reduces the dimension of the data (wrt other deterministic aggregations, such as according to input/output routers [17]); moreover, the use of reversible sketches [20] permits to trace back the flows responsible for the anomalies.…”

“…The proposed IDS can be seen as a multidimensional generalization of the scheme originally proposed in [1]. Indeed, it consists of a multidimensional sketch followed by a vectorial decision algorithm.…”

“…Roughly speaking, different types of attack cause anomalies in different traffic parameters; therefore, a wider set of traffic descriptors gives the chance of revealing more types of intrusions. "Classical" (or "monodimensional") IDSs [2], [3], [1] analyze the observed parameters separately, performing the same algorithm several times and thus increasing the computational time in order to get a more comprehensive search. The goal of this paper is to investigate the performance of "multidimensional" IDSs, which apply a single vectorial algorithm to all the traffic descriptors at the same time, leading to a processing time which is almost independent from the number of analyzed features.…”

“…The goal of this paper is to investigate the performance of "multidimensional" IDSs, which apply a single vectorial algorithm to all the traffic descriptors at the same time, leading to a processing time which is almost independent from the number of analyzed features. In more detail, we proposed a new multidimensional algorithm, which can be seen as a vectorial extension of the well-known CuSum [1] method, and compared it with five multidimensional statistical tests known in the literature (e-distance [5], Mahalanobis distance, MB-GT [6], density test [7], Log-likelihood ratio [8]). Another innovative feature of our IDS is the adaptation of the sketch to vectorial values, in order to take advantage of random aggregations in term of scalability (possibility of employing the algorithm at backbone level) and detection performance.…”