Anomalous Payload-Based Network Intrusion Detection

Abstract

We present a payload-based anomaly detector, we call PAYL, for intrusion detection. PAYL models the normal application payload of network traffic in a fully automatic, unsupervised and very effecient fashion. We first compute during a training phase a profile byte frequency distribution and their standard deviation of the application payload flowing to a single host and port. We then use Mahalanobis distance during the detection phase to calculate the similarity of new data against the pre-computed profile. The detector compares this measure against a threshold and generates an alert when the distance of the new input exceeds this threshold. We demonstrate the surprising effectiveness of the method on the 1999 DARPA IDS dataset and a live dataset we collected on the Columbia CS department network. In once case nearly 100% accuracy is achieved with 0.1% false positive rate for port 80 traffic.

Adaptive Hybrid Model for Network Intrusion Detection and Comparison among Machine Learning Algorithms

“…Past researches reveal that anomaly based intrusion detection has several flaws, namely a high false positive rate and misjudging a correct data packet as attack packet. PAYL [20] and MCPAD [21] have tried to address these issues. In this research we will also focus on reducing these issues using both supervised and unsupervised anomaly-based intrusion detection.…”

A Probabilistic Diffusion Scheme for Anomaly Detection on Smartphones

“…(b) an analysis of the eigenvalue spectrum of the vectors in the training set revealed that almost all the energy was contained in a six dimensional subspace; training and test samples were therefore projected to that subspace and the minimal l 2 norm between the current observation vector and the set of training prototypes was computed in the lower dimensional space; (c) instead of the l 2 norm, we computed the Mahalanobis distances (since it takes into account higher moments, it is frequently considered the method of choice in present day anomaly detection systems [25,26]); again, distance computation was carried out in in the low dimensional subspace; (d) self organizing maps (SOMs) were fitted to the six-dimensional approximations of the training data; the minimal distances of test vectors were computed w.r.t. the weights of the SOM neurons; (e) our algorithm presented in the previous section was applied; training and classification were based on feature vectors in R 40 .…”

“…Consequently, anomaly-based detection algorithms usually require a training (learning) phase before the actual detection (monitoring). Over the years, a large number of anomaly detection schemes have been proposed, most of them relying on a variety of machine learning and data mining techniques [19,25,26]. Decentralized malware detection schemes as well as deployment of filtering mechanisms have been studied in [6,18,24,4,5].…”

A Fast Static Analysis Approach to Detect Exploit Code Inside Network Flows

“…Anti-honeypot technology [23] Statistics-Based Payload Detection [38] Normal traffic has different byte-level statistics than worm infested traffic Blend into normal traffic [22] the aim of distinguishing data and program-like code. In this regard, we answer the following two questions.…”

“…We do not focus on the return address component and changes in software do not impact our approach. Wang et al [38] proposed a payload based anomaly detection system called PAYL which works by first training with normal network flow traffic and subsequently using several byte-level statistical measures to detect exploit code. But it is possible to evade detection by implementing the exploit code in such a way that it statistically mimics normal traffic [22].…”