Unsupervised Anomaly Detection Based on Minimum Spanning Tree Approximated Distance Measures and its Application to Hydropower Turbines

Ahmed, Imtiaz; Dagnino, Aldo; Ding, Yu

doi:10.1109/tase.2018.2848198

Cited by 38 publications

(24 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The number of the nearest neighbors ( k ) is required to be set in non-NMF-based methods. According to the guideline of [ 62 ], this paper tunes the values of k from 1 to 100 and the best value will be chosen. In our experiment, we only report the true positive detection number of all of the test methods.…”

Section: Experiments Results and Analysismentioning

confidence: 99%

Graph Regularized Deep Sparse Representation for Unsupervised Anomaly Detection

Lai

Jiang

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Anomaly detection (AD) aims to distinguish the data points that are inconsistent with the overall pattern of the data. Recently, unsupervised anomaly detection methods have aroused huge attention. Among these methods, feature representation (FR) plays an important role, which can directly affect the performance of anomaly detection. Sparse representation (SR) can be regarded as one of matrix factorization (MF) methods, which is a powerful tool for FR. However, there are some limitations in the original SR. On the one hand, it just learns the shallow feature representations, which leads to the poor performance for anomaly detection. On the other hand, the local geometry structure information of data is ignored. To address these shortcomings, a graph regularized deep sparse representation (GRDSR) approach is proposed for unsupervised anomaly detection in this work. In GRDSR, a deep representation framework is first designed by extending the single layer MF to a multilayer MF for extracting hierarchical structure from the original data. Next, a graph regularization term is introduced to capture the intrinsic local geometric structure information of the original data during the process of FR, making the deep features preserve the neighborhood relationship well. Then, a L1-norm-based sparsity constraint is added to enhance the discriminant ability of the deep features. Finally, a reconstruction error is applied to distinguish anomalies. In order to demonstrate the effectiveness of the proposed approach, we conduct extensive experiments on ten datasets. Compared with the state-of-the-art methods, the proposed approach can achieve the best performance.

show abstract

Section: Experiments Results and Analysismentioning

confidence: 99%

Graph Regularized Deep Sparse Representation for Unsupervised Anomaly Detection

Lai

Jiang

et al. 2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Using this option, we in this work make use of two existing anomaly detection approaches. One is the local minimum spanning tree (LoMST) [30] and another is the connectivity outlier factor (COF) [31].…”

Section: Anomaly Detectionmentioning

confidence: 99%

Graph Regularized Autoencoder and its Application in Unsupervised Anomaly Detection

Ahmed

Galoppo

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Dimensionality reduction is a crucial first step for many unsupervised learning tasks including anomaly detection and clustering. Autoencoder is a popular mechanism to accomplish dimensionality reduction. In order to make dimensionality reduction effective for high-dimensional data embedding nonlinear low-dimensional manifold, it is understood that some sort of geodesic distance metric should be used to discriminate the data samples. Inspired by the success of geodesic distance approximators such as ISOMAP, we propose to use a minimum spanning tree (MST), a graph-based algorithm, to approximate the local neighborhood structure and generate structure-preserving distances among data points. We use this MST-based distance metric to replace the Euclidean distance metric in the embedding function of autoencoders and develop a new graph regularized autoencoder, which outperforms a wide range of alternative methods over 20 benchmark anomaly detection datasets. We further incorporate the MST regularizer into two generative adversarial networks and find that using the MST regularizer improves the performance of anomaly detection substantially for both generative adversarial networks. We also test our MST regularized autoencoder on two datasets in a clustering application and witness its superior performance as well.

show abstract

“…Kim et al [ 41 ] applied a combination of four outlier detection methods, namely (a) Gaussian density estimation, (b) Parzen window density estimation, (c) Principal component analysis, and (d) K-means clustering to identify malicious activities in an institution using user log database. The outlier identification methods can be broadly categorized into statistical-based [ 42 ], distance-based [ 43 ], graph-based [ 44 ], clustering-based [ 45 ], density-based [ 46 ], and ensemble-based [ 47 ]. Once the outliers are detected, it is crucial to delete, consider, or modify the outlier.…”

Section: Related Workmentioning

confidence: 99%

Statistics-Based Outlier Detection and Correction Method for Amazon Customer Reviews

Chatterjee

Zhou

Abusorrah

et al. 2021

Entropy

View full text Add to dashboard Cite

People nowadays use the internet to project their assessments, impressions, ideas, and observations about various subjects or products on numerous social networking sites. These sites serve as a great source to gather data for data analytics, sentiment analysis, natural language processing, etc. Conventionally, the true sentiment of a customer review matches its corresponding star rating. There are exceptions when the star rating of a review is opposite to its true nature. These are labeled as the outliers in a dataset in this work. The state-of-the-art methods for anomaly detection involve manual searching, predefined rules, or traditional machine learning techniques to detect such instances. This paper conducts a sentiment analysis and outlier detection case study for Amazon customer reviews, and it proposes a statistics-based outlier detection and correction method (SODCM), which helps identify such reviews and rectify their star ratings to enhance the performance of a sentiment analysis algorithm without any data loss. This paper focuses on performing SODCM in datasets containing customer reviews of various products, which are (a) scraped from Amazon.com and (b) publicly available. The paper also studies the dataset and concludes the effect of SODCM on the performance of a sentiment analysis algorithm. The results exhibit that SODCM achieves higher accuracy and recall percentage than other state-of-the-art anomaly detection algorithms.

show abstract

Unsupervised Anomaly Detection Based on Minimum Spanning Tree Approximated Distance Measures and its Application to Hydropower Turbines

Cited by 38 publications

References 41 publications

Graph Regularized Deep Sparse Representation for Unsupervised Anomaly Detection

Graph Regularized Deep Sparse Representation for Unsupervised Anomaly Detection

Graph Regularized Autoencoder and its Application in Unsupervised Anomaly Detection

Statistics-Based Outlier Detection and Correction Method for Amazon Customer Reviews

Contact Info

Product

Resources

About