Unsupervised Anomaly Detection Based on Deep Autoencoding and Clustering

Abstract

The unsupervised anomaly detection task based on high-dimensional or multidimensional data occupies a very important position in the field of machine learning and industrial applications; especially in the aspect of network security, the anomaly detection of network data is particularly important. The key to anomaly detection is density estimation. Although the methods of dimension reduction and density estimation have made great progress in recent years, most dimension reduction methods are difficult to retain the key information of original data or multidimensional data. Recent studies have shown that the deep autoencoder (DAE) can solve this problem well. In order to improve the performance of unsupervised anomaly detection, we propose an anomaly detection scheme based on a deep autoencoder (DAE) and clustering methods. The deep autoencoder is trained to learn the compressed representation of the input data and then feed it to clustering approach. This scheme makes full use of the advantages of the deep autoencoder (DAE) to generate low-dimensional representation and reconstruction errors for the input high-dimensional or multidimensional data and uses them to reconstruct the input samples. The proposed scheme could eliminate redundant information contained in the data, improve performance of clustering methods in identifying abnormal samples, and reduce the amount of calculation. To verify the effectiveness of the proposed scheme, massive comparison experiments have been conducted with traditional dimension reduction algorithms and clustering methods. The results of experiments demonstrate that, in most cases, the proposed scheme outperforms the traditional dimension reduction algorithms with different clustering methods.

Scalable Learning Framework for Detecting New Types of Twitter Spam with Misuse and Anomaly Detection

“…Thus, if the difference between the input and output exceeds a certain threshold, the input can be considered to be abnormal data. Because autoencoders do not require separate labeling and can automatically distinguish between normal and abnormal data, they are widely used for anomaly detection in various fields [55][56][57][58][59]. This study leverages these autoencoder properties to detect unknown spam tweets.…”

“…In the autoencoder, multiple layers are often used for anomaly detection because this approach handles high-dimensional data effectively [57]. This study uses a multilayerbased autoencoder to increase the detection rate of unknown spam in the second stage, as illustrated in Figure 1.…”

Unsupervised Outlier Detection for Time-Series Data of Indoor Air Quality Using LSTM Autoencoder with Ensemble Method

“…In general, clustering is performed on pure input data without any processing procedures (Li et al, 2021). However, this results in poor performance in terms of explanatory power and compression because it does not reflect the high-level characteristics derived by the model (Zhang et al, 2021). Therefore, we performed clustering on the latent features extracted from LSTM-AE.…”

Monitoring of Tool and Component Wear for Self-Adaptive Digital Twins: A Multi-Stage Approach through Anomaly Detection and Wear Cycle Analysis

“…Furthermore, since duration plays a minor role in the learning phase, DBSCAN can be used [47]. As shown in [48], DBSCAN is well-fitted for anomaly detection and previous work also shows the capability in such use cases [49]. The cutting of the individual cycles takes place independently of the training and the inline application phase.…”