Collective Anomaly Detection Based on Long Short-Term Memory Recurrent Neural Networks

Abstract

Intrusion detection for computer network systems becomes one of the most critical tasks for network administrators today. It has an important role for organizations, governments and our society due to its valuable resources on computer networks. Traditional misuse detection strategies are unable to detect new and unknown intrusion. Besides, anomaly detection in network security is aim to distinguish between illegal or malicious events and normal behavior of network systems. Anomaly detection can be considered as a classification problem where it builds models of normal network behavior, which it uses to detect new patterns that significantly deviate from the model. Most of the current research on anomaly detection is based on the learning of normally and anomaly behaviors. They do not take into account the previous, recent events to detect the new incoming one. In this paper, we propose a real time collective anomaly detection model based on neural network learning and feature operating. Normally a Long Short-Term Memory Recurrent Neural Network (LSTM RNN) is trained only on normal data and it is capable of predicting several time steps ahead of an input. In our approach, a LSTM RNN is trained with normal time series data before performing a live prediction for each time step. Instead of considering each time step separately, the observation of prediction errors from a certain number of time steps is now proposed as a new idea for detecting collective anomalies. The prediction errors from a number of the latest time steps above a threshold will indicate a collective anomaly. The model is built on a time series version of the KDD 1999 dataset. The experiments demonstrate that it is possible to offer reliable and efficient for collective anomaly detection.

Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding

“…The inherent properties of LSTMs makes them an ideal candidate for anomaly detection tasks involving time-series, non-linear numeric streams of data. LSTMs are capable of learning the relationship between past data values and current data values and representing that relationship in the form of learned weights [5,21]. When trained on nominal data, LSTMs can capture and model normal behavior of a system [5], providing practitioners with a model of system behavior under normal conditions.…”

“…LSTMs are capable of learning the relationship between past data values and current data values and representing that relationship in the form of learned weights [5,21]. When trained on nominal data, LSTMs can capture and model normal behavior of a system [5], providing practitioners with a model of system behavior under normal conditions. They can also handle multivariate time-series data without the need for dimensionality reduction [33] or domain knowledge of the specific application [44], allowing for generalizability across different types of spacecraft and application domains.…”

RePAD: Real-Time Proactive Anomaly Detection for Time Series

“…However, if the training data is not representative, Greenhouse might not be able to capture pattern changes in real-world time series. Other examples based on semi-supervised learning include [7] [8].…”

Deep Learning for System Trace Restoration

“…While numerous studies discuss the use of deep learning for directly solving problems like anomaly detection [17], [18], [19], [3] and intrusion detection [20], [21], [22], only very few explicitly focus on the critical intermediate problem of restoring lossy or noisy data that is critical for correct behavior of aforementioned algorithms. Often, missing data are considered unusable and is removed completely in preprocessing to avoid having to address it.…”