In this paper, we present a deep neural network model to enhance the intrusion detection performance. A deep learning architecture combining convolution neural network and long short-term memory learns spatial-temporal features of network flows automatically. Flow features are extracted from raw network traffic captures, flows are grouped, and the consecutive N flow records are transformed into a two-dimensional array like an image. These constructed two-dimensional feature vectors are normalized and forwarded to the deep learning model. Transformation of flow information assures deep learning in a computationally efficient manner. Overall, convolution neural network learns spatial features, and long short-term memory learns temporal features from a sequence of network raw data packets. To maximize the detection performance of the deep neural network and to reach at the highest statistical metric values, we apply the tree-structured Parzen estimator seeking the optimum parameters in the parameter hyper-plane. Furthermore, we investigate the impact of flow status interval, flow window size, convolution filter size, and long short-term memory units to the detection performance in terms of level in statistical metric values. The presented flow-based intrusion method outperforms other publicly available methods, and it detects abnormal traffic with 99.09% accuracy and 0.0227 false alarm rate.
INTRODUCTIONNetwork security requires automatic detection of network attacks and learning capability to maintain the integrity and operation of the network. To detect and react to network security attacks, a well trained and tested intrusion detection system (IDS) is mandatory. The IDS systems are classified into two groups: network-based IDS, which monitors network traffic to detect intrusion attempt, and host-based IDS, which analyzes system and software logs, file system, disk activities, etc, to identify abnormal behavior. Then, misuse detection and anomaly detection identify attacks by matching patterns of known instructions and deviations from normal traffic, respectively. Generally, these two groups adopt different approaches and are used in harmony to ensure security.Along with the increasing levels of connectivity including the adoption of cloud-based services and Internet of Things, web-based services, the volume of network data is drastically growing. Former rule-based systems cannot respond to massive network data, and they are not responsive to newly created attacks based on the former rules identified or updated by security analysts. Security hole without available fix or signature is a major threat to the network. Machine learning algorithms are an intermediate solution at identifying types of attacks, and the expertise of security analysts or system Int J Network Mgmt. 2019;29:e2050.wileyonlinelibrary.com/journal/nem