Variational autoencoder-based outlier detection for high-dimensional data

Li, Yongmou; Wang, Yijie; Ma, Xingkong

doi:10.3233/ida-184240

Cited by 9 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The simulation experiments on the dataset show that this framework outperforms the current state-of-the-art outlier detection methods. Li et al [37] introduced an outlier detection method based on variational autoencoder (VAE), which combines low-dimensional representations with reconstruction errors to detect outliers. Experimental results demonstrate that the proposed method performs better than or at least comparable to existing methods.…”

Section: Outlier Detection Modelmentioning

confidence: 99%

Similarity Distribution Density: An Optimized Approach to Outlier Detection

Quan,

Gong,

Jiang

2023

Electronics

View full text Add to dashboard Cite

When dealing with uncertain data, traditional model construction methods often ignore or filter out noise data to improve model performance. However, this simple approach can lead to insufficient data utilization, model bias, reduced detection ability, and decreased robustness of detection models. Outliers can be considered as data that are inconsistent with other patterns at certain specific moments and are not always negative data, so their emergence is not always bad. In the process of data analysis, outliers play a crucial role in sample vector recognition, missing value processing, and model stability verification. In addition, unsupervised models have very high computation costs when recognizing outliers, especially non-parameterized unsupervised models. To solve the above problems, we used semi-supervised learning processes and used similarity as a negative selection criterion to propose a local density verification detection model (Vd-LOD). This model establishes similarity pseudo-labels for multi-label and multi-type samples, verifies the accuracy of outlier values based on local outlier factors, and increases the detector’s sensitivity to outliers. The experimental results show that under different parameter settings with varying outlier quantities, Vd-LOD outperforms other detection models in terms of the significant increase in average time consumption caused by verifying the presence of relationships, while also achieving an approximate 6% improvement in average detection accuracy.

show abstract

Section: Outlier Detection Modelmentioning

confidence: 99%

Similarity Distribution Density: An Optimized Approach to Outlier Detection

Quan,

Gong,

Jiang

2023

Electronics

View full text Add to dashboard Cite

show abstract

“…These outlier detection methods focus on subspace selection, data dimensionality reduction, and reconstruction. The methods can be categorized into two types: feature selection-based and feature transformation-based [19]. The feature selection method, also referred to as subspace outlier detection [20], is based on detecting outliers in a certain feature subset.…”

Section: Related Workmentioning

confidence: 99%

An Auto-Encoder with Genetic Algorithm for High Dimensional Data: Towards Accurate and Interpretable Outlier Detection

Zhang

Bah

et al. 2022

Algorithms

View full text Add to dashboard Cite

When dealing with high-dimensional data, such as in biometric, e-commerce, or industrial applications, it is extremely hard to capture the abnormalities in full space due to the curse of dimensionality. Furthermore, it is becoming increasingly complicated but essential to provide interpretations for outlier detection results in high-dimensional space as a consequence of the large number of features. To alleviate these issues, we propose a new model based on a Variational AutoEncoder and Genetic Algorithm (VAEGA) for detecting outliers in subspaces of high-dimensional data. The proposed model employs a neural network to create a probabilistic dimensionality reduction variational autoencoder (VAE) that applies its low-dimensional hidden space to characterize the high-dimensional inputs. Then, the hidden vector is sampled randomly from the hidden space to reconstruct the data so that it closely matches the input data. The reconstruction error is then computed to determine an outlier score, and samples exceeding the threshold are tentatively identified as outliers. In the second step, a genetic algorithm (GA) is used as a basis for examining and analyzing the abnormal subspace of the outlier set obtained by the VAE layer. After encoding the outlier dataset’s subspaces, the degree of anomaly for the detected subspaces is calculated using the redefined fitness function. Finally, the abnormal subspace is calculated for the detected point by selecting the subspace with the highest degree of anomaly. The clustering of abnormal subspaces helps filter outliers that are mislabeled (false positives), and the VAE layer adjusts the network weights based on the false positives. When compared to other methods using five public datasets, the VAEGA outlier detection model results are highly interpretable and outperform or have competitive performance compared to current contemporary methods.

show abstract

“…For example, Zong et al combined the Autoencoder with gaussian mixture model to jointly consider reconstruction error and the distribution of intermediate hidden layer's variables for anomaly detection [25]. Li et al combined the intermediate layer of a Variational Autoencoder with the reconstructed error for anomaly detection [13]. DNNs large-scale parameters and iterative optimization enable the models to obtain accurate representations by learning from deep relationships and patterns automatically in training data.…”

Section: Introductionmentioning

confidence: 99%

An anomaly detection method based on random convolutional kernel and isolation forest for equipment state monitoring

Shu¹,

Zhang²,

Li³

et al. 2022

Eksploatacja i Niezawodność – Maintenance and Reliability

View full text Add to dashboard Cite

Anomaly detection plays an essential role in health monitoring and reliability assurance of complex system. However, previous researches suffer from distraction by outliers in training and extensively relying on empiric-based feature engineering, leading to many limitations in the practical application of detection methods. In this paper, we propose an unsupervised anomaly detection method that combines random convolution kernels with isolation forest to tackle the above problems in equipment state monitoring. The random convolution kernels are applied to generate cross-dimensional and multi-scale features for multi-dimensional time series, with combining the time series decomposing method to select abnormally sensitive features for automatic feature extraction. Then, anomaly detection is performed on the obtained features using isolation forests with low requirements for purity of training sample. The verification and comparison on different types of datasets show the performance of the proposed method surpass the traditional methods in accuracy and applicability.

show abstract

Variational autoencoder-based outlier detection for high-dimensional data

Cited by 9 publications

References 14 publications

Similarity Distribution Density: An Optimized Approach to Outlier Detection

Similarity Distribution Density: An Optimized Approach to Outlier Detection

An Auto-Encoder with Genetic Algorithm for High Dimensional Data: Towards Accurate and Interpretable Outlier Detection

An anomaly detection method based on random convolutional kernel and isolation forest for equipment state monitoring

Contact Info

Product

Resources

About