A Scalable and Hybrid Intrusion Detection System Based on the Convolutional-LSTM Network

Abstract

With the rapid advancements of ubiquitous information and communication technologies, a large number of trustworthy online systems and services have been deployed. However, cybersecurity threats are still mounting. An intrusion detection (ID) system can play a significant role in detecting such security threats. Thus, developing an intelligent and accurate ID system is a non-trivial research problem. Existing ID systems that are typically used in traditional network intrusion detection system often fail and cannot detect many known and new security threats, largely because those approaches are based on classical machine learning methods that provide less focus on accurate feature selection and classification. Consequently, many known signatures from the attack traffic remain unidentifiable and become latent. Furthermore, since a massive network infrastructure can produce large-scale data, these approaches often fail to handle them flexibly, hence are not scalable. To address these issues and improve the accuracy and scalability, we propose a scalable and hybrid IDS, which is based on Spark ML and the convolutional-LSTM (Conv-LSTM) network. This IDS is a two-stage ID system: the first stage employs the anomaly detection module, which is based on Spark ML. The second stage acts as a misuse detection module, which is based on the Conv-LSTM network, such that both global and local latent threat signatures can be addressed. Evaluations of several baseline models in the ISCX-UNB dataset show that our hybrid IDS can identify network misuses accurately in 97.29% of cases and outperforms state-of-the-art approaches during 10-fold cross-validation tests.

Building an efficient intrusion detection system based on feature selection and ensemble classifier

“…A comparative analysis performed on several IDS datasets has proven that IG-PCA-Ensemble method exhibits better performance than the majority of existing approaches. Due to large-scale data produced from a massive network infrastructure, Khan et al [45] proposed a scalable and hybrid IDS, which is based on Spark ML and Convolutional-LSTM (Conv-LSTM) network to employ the anomaly and misuse detection separately. Zhong et al [99] also proposed a new anomaly detection model called HELAD, which is based on the Damped Incremental Statistics algorithm for feature selection and organic integration of multiple deep learning techniques for classification.…”

Artificial Intelligence in the Cyber Domain: Offense and Defense

“…Another approach [36] presented a hybrid IDS utilizing spark ML and the convolutional-LSTM network. The ISCX-UNB dataset was used to evaluate the performance of the method.…”

“…First, the combination of several AI-based techniques in a defense solution may still an interesting research direction. For example, the incorporation of bio-inspired computation and ML/DL approaches shows promising results in malware detection [18][19][20][21][22] or [36][37][38] for detecting the network intrusion. Hence, the combination of these two techniques is a very potential research direction due to the number of bio-inspired algorithms exploited in cybersecurity still being limited.…”

Hierarchical Intrusion Detection Using Machine Learning and Knowledge Model

“…As they usually gain better performance than traditional machine learning models, they became popular methods in this area, although their explanatory capacity is often very limited. Numerous different approaches are used, including both shallow and deep learning methods [2], or a combination of both [3].…”