Pattern Based Network Security Using Semi-supervised Learning

Abstract

Network security is becoming increasingly important in today's internetworked systems. With the development of internet, its use on public networks, the number and the severity of security threats has increased significantly. Intrusion Detection System can provide a layer of security to these systems. The goal of intrusion detection system is to identify entities who attempt to subvert in-place security controls. The field of machine learning is gaining increasing attention in the development of intrusion detection systems. The machine learning techniques used for solving intrusion detection problem can be broadly classified into three broad categories: Unsupervised, supervised and semi-supervised. The supervised learning method exhibits good classification accuracy for known attacks. But it requires large amount of training data. In real world the availability of labeled data is time consuming and costly. An emerging field of semisupervised learning offers a promising direction for further research. So in this work we propose a semi-supervised approach for pattern based IDS to improve performance of supervised approach. The experimentation is performed on KDD CUP99 dataset.

Classification of Partially Labeled Malicious Web Traffic in the Presence of Concept Drift

“…The first five papers [3], [5], [12], [17], [20] considered the intrusion detection problem, whereas the sixth paper [19] focused on determining the optimal amount of labeled data needed for good classification of normal and malicious executable files. The work in [5] used the DARPA dataset with semi-supervised learning to build an alert filter for intrusion detection.…”

“…With 10% partially labeled data semi-supervised learning was better in detecting true attacks and reducing false alarms than supervised learning. Two papers [3] and [17] used the KDD 1999 Cup dataset, which is a derivative of the DARPA dataset, with semi-supervised learning to distinguish among normal and malicious classes. A semi-supervised approach to anomaly and misuse detection using partially observable Markov decision process was proposed in [12].…”

“…First, none of the related works was based on only malicious data collected by advertised, fully functional, threetier honeypot system, with standard security settings. Most of the papers [1], [2], [3], [4], [5], [14], [15], [16], [17] used the outdated DARPA and KDD 1999 datasets, which in addition to the normal traffic include four classes of simulated attacks.…”

“…This classification is important because attacks are much more critical events than vulnerability scans. Other works distinguished among one normal and four malicious classes [1], [2], [3], [4], [5], [14], [15], [16], [17], or among one normal and one malicious class [12], [20], [19].…”

“…Finally, we study both concept drift and concept evolution whereas some of the related work papers did not study either of them [3], [5], [12], [17], [19], [20], or studied only concept drift [1], [2], [4], [15], [16] or only concept evolution [14]. standard deviation of the number of parameters passed to application within a session; boolean indications of whether: (35) robots.txt file was accessed in any request of that session; (36) it was a night session (from 12 am to 8 am local time); (37) there was a remote site injection in at least one request; (38) a semicolon was used to divide multiple attributes passed to an application in at least one request; (39) a string containing suspicious encoding in any of the requests; (40) a reserved character was used in any of the requests; (41) an ASCII control character was used in any of the requests; (42) a non-ASCII control character was used in any of the requests; and (43) an invalid encoding was used in any of the requests.…”