Mining distance-based outliers in near linear time with randomization and a simple pruning rule

Abstract

Defining outliers by their distance to neighboring examples is a popular approach to finding unusual examples in a data set. Recently, much work has been conducted with the goal of finding fast algorithms for this task. We show that a simple nested loop algorithm that in the worst case is quadratic can give near linear time performance when the data is in random order and a simple pruning rule is used. We test our algorithm on real high-dimensional data sets with millions of examples and show that the near linear scaling holds over several orders of magnitude. Our average case analysis suggests that much of the efficiency is because the time to process non-outliers, which are the majority of examples, does not depend on the size of the data set..

Efficient Pruning Schemes for Distance-Based Outlier Detection

“…This gives a summary classification of all existing detection techniques. -We demonstrate a huge improvement in execution time by using multiple pruning rules in two phases, compared with outstanding existing nested-loop distance-based methods, ORCA [11] and RBRP [12]. Since ORCA, RBRP and MIRO use the same notion of outlier (Section 2), outliers identified by the three techniques are exactly the same.…”

“…Here two pruning rules are utilized: a) first triangular inequality on the data point's outlier score is used, and then b) the outlier score is compared with the minimum score required to be an outlier. The second check is similar to that of ORCA [11]. However, while ORCA starts with a cutoff of 0, in MIRO the initial cutoff is obtained from the first phase, and hence converges faster.…”

“…Among these, approaches for pruning the outlier search space and distance computation reduction tech-niques are dominant. Computation reduction approaches [7,12,11,6] usually fix the desired number of outliers to a certain value (e.g., top n outliers), and deploy data structures similar to those used in Ramaswamy's index-based algorithm.…”

“…This leads to high execution times and has motivated many attempts to produce efficient algorithms to mine outliers. Among them, outstanding work by Bay and Schwabacher [11] and Ghoting et al [12] aim to reduce execution time by utilizing a simple pruning nested-loop algorithm.…”

“…In this work, we employ a global outlier function based on [6], although the ideas employed in MIRO can also be adapted to use other functions. The intuition and quality of detection results of the chosen outlier definition are based on solid foundations as shown by prior work [6,11]. This definition is also employed in other popular techniques on outlier detection [12].…”