Anomalous Network Packet Detection Using Data Stream Mining

Abstract

In recent years, significant research has been devoted to the development of Intrusion Detection Systems (IDS) able to detect anomalous computer network traffic indicative of malicious activity. While signature-based IDS have proven effective in discovering known attacks, anomaly-based IDS hold the even greater promise of being able to automatically detect previously undocumented threats. Traditional IDS are generally trained in batch mode, and therefore cannot adapt to evolving network data streams in real time. To resolve this limitation, data stream mining techniques can be utilized to create a new type of IDS able to dynamically model a stream of network traffic. In this paper, we present two methods for anomalous network packet detection based on the data stream mining paradigm. The first of these is an adapted version of the DenStream algorithm for stream clustering specifically tailored to evaluate network traffic. In this algorithm, individual packets are treated as points and are flagged as normal or abnormal based on their belonging to either normal or outlier clusters. The second algorithm utilizes a histogram to create a model of the evolving network traffic to which incoming traffic can be compared using Pearson correlation. Both of these algorithms were tested using the first week of data from the DARPA ’99 dataset with Generic HTTP, Shell-code and Polymorphic attacks inserted. We were able to achieve reasonably high detection rates with moderately low false positive percentages for different types of attacks, though detection rates varied between the two algorithms. Overall, the histogram-based detection algorithm achieved slightly superior results, but required more parameters than the clustering-based algorithm. As a result of its fewer parameter requirements, the clustering approach can be more easily generalized to different types of network traffic streams

Data Stream Subspace Clustering for Anomalous Network Packet Detection

“…For all days except for Thursday, the StreamPreDeCon clustering algorithm achieved the highest detection rates with the least false positive rates for the Shell-code attacks averaging 94 percent with the Because the CLET attacks are meant to fool the anomaly-based IDS through polymorphic techniques, these attacks cause the lowest acceptable detection rates. Despite the slightly poor detection of CLET attacks, the StreamPreDeCon IDS on average had mostly higher sensitivity values with substantially lower false positive rates than the results of [10].…”

“…Tested on a dataset comprised of normal packets from the first week of the DARPA '99 intrusion detection evaluation dataset and various types of malicious traffic from [1], the IDS based on StreamPreDeCon out-performed previous stream-based IDS [10] using the same dataset in all days except for one day. For these days, the anomalous packet detection of the StreamPreDeCon IDS improved the sensitivity rate of the DenStream based IDS from 30% -90% to 60% -94% and reduced the false positive rates from a high of 20% to between 1% and 10%.…”

“…In 2011 we created two anomaly-based IDSs based on stream mining algorithms, and tested the IDSs on network packets represented by 2-gram features [10]. The first IDS used a modification of the density-based stream clustering algorithm DenStream [11].…”

Minimal Triangle Area Mahalanobis Distance for Stream Homogeneous Group-based DDoS Classification

“…The first process was traffic features selection, which selects proper features for system input. In the second process, we construct a homogeneous group by adopting the micro-clustering approach in Denstream [12]. The stream traffic was grouped according to the minimal distance to existing group profile and processed further when the group size is adequate for group-based classification input.…”

“…Aggregate traffic features are commonly proposed to detect traffic anomaly, such as [6,7,8,9]. By the used of machine learning such as clustering method, the system can determine clusters of traffic such in [10,11,12], but the output has no concern about the types of an anomaly of the formed groups. The classification has better used in security system such in [13,14,15], a system can single out the specific anomalous packet or connection and determine the known types of anomaly.…”