Highly Accurate and Efficient Two Phase - Intrusion Detection System (TP-IDS) using Distributed Processing of HADOOP & Machine Learning Techniques

Abstract

Network security and data security are the biggest concerns now a days. Every organization decides their future business process based on the past and day to day transactional data. This data may consist of consumers confidential data, which needs to be kept secure. Also, the network connections when established with the external communication devices or entities, a care should be taken to authenticate these and block the unwanted access. This consists of identification of the malicious connection nodes or identification of normal connection nodes. We expect, everytime whenever there is a connection request, it should be recognized as a type of normal node or malicious node connection request. For that, we use a continuous monitoring of the network input traffic to recognize the malicious connection request called as an intrusion and this type of monitoring system is called as Intrusion detection system(IDS). IDS helps us to protect our network and data from insecure and malicious network connections. Many such systems exists in the real time scenario, but they have critical issues of performance like accuracy and efficiency. These issues are addressed as a part of this research work of IDS using machine learning techniques. The TP-IDS is designed in two phases for increasing accuracy. In phase I of TP-IDS, Suppor Vector Machine (SVM) and k Nearest Neighbor (kNN) are used. In phase II of TP-IDS, Decision Tree (DT) and Naïve Bayes (NB) are used, where phase II is the validation phase of the system for increasing accuracy. Also, both the phases are having Hadoop distributed file system underlying data storage & processing architecture, which allows parallel processing to increase the speed of the system and hence achieve the efficiency in TP-IDS.