A Comparison of Outlier Detection Techniques for High-Dimensional Data

Xu, Xinglian; Liu, Huawen; Li, Li; Yao, Minghai

doi:10.2991/ijcis.11.1.50

Cited by 38 publications

(20 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Anomaly detection has been the major focus of several kinds of research and scientific papers over the years and according to Xu et al in [14], Anomaly detection is a hot topic in terms of machine learning and its increasing greatly, being applied in a wide range of fields while it plays a great role in several other domains. Therefore, in the existing literature, several techniques and approaches have been proposed to detect anomalies as well as to improve the performance of existing anomaly detection techniques.…”

Section: Related Workmentioning

confidence: 99%

“…In [14], several techniques have been surveyed and the main focus was on comparing the techniques for unlabeled highdimensional benchmark datasets. The authors faced a challenge in identifying the threshold between anomaly and non-anomaly data points and another challenge in choosing the best features in this high-dimensional space.…”

Section: A Anomaly Detectionmentioning

confidence: 99%

“…Similarly, the authors in [15] used a large scale data with a high-dimensional space. They also faced the same challenges as in [14]. However, they investigated different types of learning models for deep anomaly detection techniques.…”

Section: A Anomaly Detectionmentioning

confidence: 99%

“…However, they investigated different types of learning models for deep anomaly detection techniques. Finally, they adopted a deep hybrid model by using one-class SVM (OC-SVM) together with deep neural networks model called One class neural network (OC-NN) to detect the anomalies and conversely to [14], the authors here used Autoencoders as feature extractors and nonetheless they used the same techniques to evaluate the model while the results did not show clear improvement.…”

Section: A Anomaly Detectionmentioning

confidence: 99%

See 3 more Smart Citations

Assembly Line Anomaly Detection and Root Cause Analysis Using Machine Learning

Abdelrahman

Keikhosrokiani

2020

IEEE Access

View full text Add to dashboard Cite

Anomaly detection is becoming widely used in Manufacturing Industry to enhance product quality. At the same time, it plays a great role in several other domains due to the fact that anomaly may reveal rare but represent an important phenomenon. The objective of this paper is to detect anomalies and identify the possible variables that caused these anomalies on historical assembly data for two series of products. Multiple anomaly detection techniques were performed; HBOS, IForest, KNN, CBLOF, OCSVM, LOF, and ABOD. Moreover, we used AUROC and Rank Power as performance metrics, followed by Boosting ensemble learning method to ensure the best anomaly detectors robustness. The techniques that gave the highest performance are KNN, ABOD for both product series datasets with 0.95 and 0.99 AUROC respectively. Finally, we applied a statistical root cause analysis on the detected anomalies with the use of Pareto chart to visualize the frequency of the possible causes and its cumulative occurrence. The results showed that there are seven rejection causes for both product series, whereas the first three causes are responsible for 85% of the rejection rates. Besides, assembly machines engineers reported a significant reduction in the rejection rates in both assembly machines after tuning the specification limits of the rejection causes identified by this research results.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: A Anomaly Detectionmentioning

confidence: 99%

Section: A Anomaly Detectionmentioning

confidence: 99%

Section: A Anomaly Detectionmentioning

confidence: 99%

See 2 more Smart Citations

Assembly Line Anomaly Detection and Root Cause Analysis Using Machine Learning

Abdelrahman

Keikhosrokiani

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Outlier detection algorithms have extensively been tackled in the past fifteen years. Many algorithms with different approaches have been introduced in the literature [4][5][6][7][8][9][10][11] which can be, in general, categorized into [12][13][14]: statistical-based [15,16], distance-based [17,18], density-based [19,20] and clustering-based methods [9,[21][22][23]. Statistical-based approaches aim at finding the probability distribution/model of the underlying normal data and define outliers as those points that do not conform to that model.…”

Section: Introductionmentioning

confidence: 99%

OFCOD: On the Fly Clustering Based Outlier Detection Framework

Elmogy

Rizk

Sarhan

2020

Data

View full text Add to dashboard Cite

In data mining, outlier detection is a major challenge as it has an important role in many applications such as medical data, image processing, fraud detection, intrusion detection, and so forth. An extensive variety of clustering based approaches have been developed to detect outliers. However they are by nature time consuming which restrict their utilization with real-time applications. Furthermore, outlier detection requests are handled one at a time, which means that each request is initiated individually with a particular set of parameters. In this paper, the first clustering based outlier detection framework, (On the Fly Clustering Based Outlier Detection (OFCOD)) is presented. OFCOD enables analysts to effectively find out outliers on time with request even within huge datasets. The proposed framework has been tested and evaluated using two real world datasets with different features and applications; one with 699 records, and another with five millions records. The experimental results show that the performance of the proposed framework outperforms other existing approaches while considering several evaluation metrics.

show abstract

A geometric framework for outlier detection in high‐dimensional data

Herrmann

Pfisterer

Scheipl

2023

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Outlier or anomaly detection is an important task in data analysis. We discuss the problem from a geometrical perspective and provide a framework which exploits the metric structure of a data set. Our approach rests on the manifold assumption, that is, that the observed, nominally high‐dimensional data lie on a much lower dimensional manifold and that this intrinsic structure can be inferred with manifold learning methods. We show that exploiting this structure significantly improves the detection of outlying observations in high dimensional data. We also suggest a novel, mathematically precise and widely applicable distinction between distributional and structural outliers based on the geometry and topology of the data manifold that clarifies conceptual ambiguities prevalent throughout the literature. Our experiments focus on functional data as one class of structured high‐dimensional data, but the framework we propose is completely general and we include image and graph data applications. Our results show that the outlier structure of high‐dimensional and non‐tabular data can be detected and visualized using manifold learning methods and quantified using standard outlier scoring methods applied to the manifold embedding vectors. This article is categorized under: Technologies > Structure Discovery and Clustering Fundamental Concepts of Data and Knowledge > Data Concepts Technologies > Visualization

show abstract

A Comparison of Outlier Detection Techniques for High-Dimensional Data

Cited by 38 publications

References 32 publications

Assembly Line Anomaly Detection and Root Cause Analysis Using Machine Learning

Assembly Line Anomaly Detection and Root Cause Analysis Using Machine Learning

OFCOD: On the Fly Clustering Based Outlier Detection Framework

A geometric framework for outlier detection in high‐dimensional data

Contact Info

Product

Resources

About