The complexity of network intrusion detection systems (IDSs) is increasing due to the continuous increases in network traffic, various attacks and the ever-changing network environment. In addition, network traffic is asymmetric with few attack data, but the attack data are so complex that it is difficult to detect one. Many studies on improving intrusion detection performance using feature engineering have been conducted. These studies work well in the dataset environment; however, it is challenging to cope with a changing network environment. This paper proposes an intrusion detection hyperparameter control system (IDHCS) that controls and trains a deep neural network (DNN) feature extractor and k-means clustering module as a reinforcement learning model based on proximal policy optimization (PPO). An IDHCS controls the DNN feature extractor to extract the most valuable features in the network environment, and identifies intrusion through k-means clustering. Through iterative learning using the PPO-based reinforcement learning model, the system is optimized to improve performance automatically according to the network environment, where the IDHCS is used. Experiments were conducted to evaluate the system performance using the CICIDS2017 and UNSW-NB15 datasets. In CICIDS2017, an F1-score of 0.96552 was achieved and UNSW-NB15 achieved an F1-score of 0.94268. An experiment was conducted by merging the two datasets to build a more extensive and complex test environment. By merging datasets, the attack types in the experiment became more diverse and their patterns became more complex. An F1-score of 0.93567 was achieved in the merged dataset, indicating 97% to 99% performance compared with CICIDS2017 and UNSW-NB15. The results reveal that the proposed IDHCS improved the performance of the IDS by automating learning new types of attacks by managing intrusion detection features regardless of the network environment changes through continuous learning.

As the Internet of Things (IoT) continues to grow, a vast amount of data is generated. The IoT environment is quite sensitive to security challenges because personal information may be leaked or sensor data may be manipulated, which could cause accidents. Because traditional intrusion detection system (IDS) studies are often designed to work well on datasets, it is unknown whether they would work well in a changing network environment. In addition, IDSs for protecting IoT environments have been studied, but their performance was verified using datasets unrelated to the IoT, so it is not known whether the performance would be effective in an IoT environment. In this study, we propose an intrusion detection hyperparameter control system (ID-HyConSys) that automates the IDS using proximal policy optimization (PPO) to solve these problems and reliably protect the IoT environment. ID-HyConSys consists of an intrusion detection module consisting of a deep neural network (DNN) feature extractor that extracts efficient features from a changing network environment, a k-means cluster that clusters the extracted data, and a PPO agent that automates the IDS through learning and control. Through experimentation, it was confirmed that the hidden layer configuration, the number of feature extractions by the DNN feature extractor, and the number of clusters in the k-means cluster significantly affected the intrusion detection performance. The PPO directly controls these hyperparameters and determines the optimized value itself. The performance of ID-HyConSys was evaluated using the CICIDS2017 and MQTTset datasets. An F1-score of 0.9707 on CICIDS2017 and an F1-score of 0.9973 on the MQTTset were obtained. Finally, we merged the two datasets and obtained an F1-score of 0.9901. The superiority of the ID-HyConSys proposed in this study was confirmed because ID-HyConSys showed high performance on each dataset and, at the same time, very high performance on complex merged datasets. ID-HyConSys is expected to protect the IoT environment more quickly and safely by automatically learning network changes and adjusting the intrusion detection module accordingly.