Fast Outlier Detection in High Dimensional Spaces

Abstract

In this paper we propose a new definition of distance-based outlier that considers for each point the sum of the distances from its k nearest neighbors, called weight. Outliers are those points having the largest values of weight. In order to compute these weights, we find the k nearest neighbors of each point in a fast and efficient way by linearizing the search space through the Hilbert space filling curve. The algorithm consists of two phases, the first provides an approximated solution, within a small factor, after executing at most d + 1 scans of the data set with a low time complexity cost, where d is the number of dimensions of the data set. During each scan the number of points candidate to belong to the solution set is sensibly reduced. The second phase returns the exact solution by doing a single scan which examines further a little fraction of the data set. Experimental results show that the algorithm always finds the exact solution during the first phase after d d + 1 steps and it scales linearly both in the dimensionality and the size of the data set.

Finding Fraud in Health Insurance Data with Two-Layer Outlier Detection Approach

“…In other words, distance can be seen as a radius around p, and if the percentage of points within this radius is smaller than (1 − perc), p is declared anomalous. A yet another definition, [2], assigns a weight to each point p, which is defined as the sum of the k-distance of all points within the k-distance neighborhood of p. Outliers are those points that have the biggest weight. There are some small differences between the three definitions given above.…”

“…For the definition by [8] it may be hard to set the parameters appropriately. The definition of [2] overcomes these problems, but is computationally expensive. We used in our experiments this later definition of the scoring function.…”

“…We standardized both dimensions. Because we were interested in global outliers we used as an entity outlier score the mean distance to the k nearest neighbors, [2]. Next, we calculated the deviation from the mean statistic per patient per drug type, and aggregated the scores by summing them.…”

“…In the first stage we calculate outlier scores of single records. We use here classical methods for outlier detection that are based on distance measures, [2], or density estimation, [5].…”

Fast mining of distance-based outliers in high-dimensional datasets

“…Such approaches do not work well in even moderately high dimensional spaces and finding the right model is often a difficult task in its own right. To overcome these limitations, researchers have turned to various non-parametric approaches that use a point's distance to its nearest neighbor as a measure of unusualness [1,10,11]. The following (among others [3]) is a popular definition of distance-based outliers:…”