Efficient Pruning Schemes for Distance-Based Outlier Detection

ACM Trans. Knowl. Discov. Data

et al. 2015

541

417

An integrated framework for density-based cluster analysis, outlier detection, and data visualization is introduced in this article. The main module consists of an algorithm to compute hierarchical estimates of the level sets of a density, following Hartigan's classic model of density-contour clusters and trees. Such an algorithm generalizes and improves existing density-based clustering techniques with respect to different aspects. It provides as a result a complete clustering hierarchy composed of all possible density-based clusters following the nonparametric model adopted, for an infinite range of density thresholds. The resulting hierarchy can be easily processed so as to provide multiple ways for data visualization and exploration. It can also be further postprocessed so that: (i) a normalized score of "outlierness" can be assigned to each data object, which unifies both the global and local perspectives of outliers into a single definition; and (ii) a "flat" (i.e., nonhierarchical) clustering solution composed of clusters extracted from local cuts through the cluster tree (possibly corresponding to different density thresholds) can be obtained, either in an unsupervised or in a semisupervised way. In the unsupervised scenario, the algorithm corresponding to this postprocessing module provides a global, optimal solution to the formal problem of maximizing the overall stability of the extracted clusters. If partially labeled objects or instance-level constraints are provided by the user, the algorithm can solve the problem by considering both constraints violations/satisfactions and cluster stability criteria. An asymptotic complexity analysis, both in terms of running time and memory space, is described. Experiments are reported that involve a variety of synthetic and real datasets, including comparisons with state-of-the-art, density-based clustering and (global and local) outlier detection methods.

Section: Outlier Detectionmentioning

confidence: 99%

Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection

Campello

Moulavi

ACM Trans. Knowl. Discov. Data

et al. 2015

541

417

“…This concept is sometimes supported by data partitioning techniques as, for example, in refs. 27,96.…”

Section: Efficiency and Effectiveness For Outlier Detection In Higmentioning

confidence: 99%

A survey on unsupervised outlier detection in high‐dimensional numerical data

Schubert

Kriegel

2012

Statistical Analysis

685

413

High‐dimensional data in Euclidean space pose special challenges to data mining algorithms. These challenges are often indiscriminately subsumed under the term ‘curse of dimensionality’, more concrete aspects being the so‐called ‘distance concentration effect’, the presence of irrelevant attributes concealing relevant information, or simply efficiency issues. In about just the last few years, the task of unsupervised outlier detection has found new specialized solutions for tackling high‐dimensional data in Euclidean space. These approaches fall under mainly two categories, namely considering or not considering subspaces (subsets of attributes) for the definition of outliers. The former are specifically addressing the presence of irrelevant attributes, the latter do consider the presence of irrelevant attributes implicitly at best but are more concerned with general issues of efficiency and effectiveness. Nevertheless, both types of specialized outlier detection algorithms tackle challenges specific to high‐dimensional data. In this survey article, we discuss some important aspects of the ‘curse of dimensionality’ in detail and survey specialized algorithms for outlier detection from both categories. © 2012 Wiley Periodicals, Inc. Statistical Analysis and Data Mining, 2012

“…In the context of outlier detection, much work therefore aimed at making outlier detection more efficient in retrieving the topk outliers, omitting the assessment of outlierness for those objects that, given the already received results, cannot rank among the top-k anymore. Examples are [2,4,19,26,36,37,44]. Usually, in these methods, the efficiency is evaluated while the quality of the ranking result is bound to be optimal w.r.t.…”

Section: Related Workmentioning

confidence: 99%

“…Consequently, outlier detection is a quite active research area with many new methods proposed every year, based on different underlying methodologies like statistical reasoning [17], distances [2,23,36,37,44,47], or densities [6,8,20,24]. An open and widely ignored problem, though, is the proper and informative evaluation of the rankings of outlier scores provided by the different methods.…”

Section: Introductionmentioning

confidence: 99%

On Evaluation of Outlier Rankings and Outlier Scores

Schubert

Wojdanowski

Proceedings of the 2012 SIAM International Conference on Data Mining

et al. 2012

131

142

Outlier detection research is currently focusing on the development of new methods and on improving the computation time for these methods. Evaluation however is rather heuristic, often considering just precision in the top k results or using the area under the ROC curve. These evaluation procedures do not allow for assessment of similarity between methods. Judging the similarity of or correlation between two rankings of outlier scores is an important question in itself but it is also an essential step towards meaningfully building outlier detection ensembles, where this aspect has been completely ignored so far. In this study, our generalized view of evaluation methods allows both to evaluate the performance of existing methods as well as to compare different methods w.r.t. their detection performance. Our new evaluation framework takes into consideration the class imbalance problem and offers new insights on similarity and redundancy of existing outlier detection methods. As a result, the design of effective ensemble methods for outlier detection is considerably enhanced.