Unsupervised Novelty Detection Using Deep Autoencoders with Density Based Clustering

Abstract

Novelty detection is a classification problem to identify abnormal patterns; therefore, it is an important task for applications such as fraud detection, fault diagnosis and disease detection. However, when there is no label that indicates normal and abnormal data, it will need expensive domain and professional knowledge, so an unsupervised novelty detection approach will be used. On the other hand, nowadays, using novelty detection on high dimensional data is a big challenge and previous research suggests approaches based on principal component analysis (PCA) and an autoencoder in order to reduce dimensionality. In this paper, we propose deep autoencoders with density based clustering (DAE-DBC); this approach calculates compressed data and error threshold from deep autoencoder model, sending the results to a density based cluster. Points that are not involved in any groups are not considered a novelty; the grouping points will be defined as a novelty group depending on the ratio of the points exceeding the error threshold. We have conducted the experiment by substituting components to show that the components of the proposed method together are more effective. As a result of the experiment, the DAE-DBC approach is more efficient; its area under the curve (AUC) is shown to be 13.5 percent higher than state-of-the-art algorithms and other versions of the proposed method that we have demonstrated.

DeepAnT: A Deep Learning Approach for Unsupervised Anomaly Detection in Time Series

“…Due to its efficient data encoding in an unsupervised manner, it is also gaining popularity for anomaly and novelty detection problems. Amarbayasgalan et al [31] proposed a novelty detection technique based on deep autoencoders. Their approach gets compressed data and error threshold from deep autoencoders and apply density-based clustering on the compressed data to get novelty groups with low density.…”

Spending Segmentation and Outlier Detection in Brazilian Elections

“…By running a k-means clustering method on a subset containing all samples above a reconstruction error threshold, [Zamini and Montazer 2018] was able to better identify outliers in a credit card fraud dataset. Another work [Amarbayasgalan et al 2018] has used the density-based clustering algorithm DBSCAN [Ester et al 1996] to find clusters in the complete dataset on the low-dimensional space, defining clusters as outliers in case their members exceeded a certain reconstruction error threshold. Although powerful to capture non-linear relationships in this dimension reduction mapping, its use may reduce the model results explainability.…”

Unsupervised hybrid anomaly detection model for logistics fleet management systems

“…The model was a combination of AE for dimensional reduction and ensemble k-nearest neighbour for clustering [18]. Another research also used AE as a base model and performed density-based spatial clustering of applications with noise to estimate density [19]. The combination of supervised and unsupervised learning was also adapted to detect denial of service (DoS), probe and normal in the hierarchical level.…”