A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes

Abstract

Outlier detection has attracted substantial attention in many applications and research areas; some of the most prominent applications are network intrusion detection or credit card fraud detection. Many of the existing approaches are based on calculating distances among the points in the dataset. These approaches cannot easily adapt to current datasets that usually contain a mix of categorical and continuous attributes, and may be distributed among different geographical locations. In addition, current datasets usually have a large number of dimensions. These datasets tend to be sparse, and traditional concepts such as Euclidean distance or nearest neighbor become unsuitable. We propose a fast distributed outlier detection strategy intended for datasets containing mixed attributes. The proposed method takes into consideration the sparseness of the dataset, and is experimentally shown to be highly scalable with the number of points and the number of attributes in the dataset. Experimental results show that the proposed outlier detection method compares very favorably with other state-of-the art outlier detection strategies proposed in the literature and that the speedup achieved by its distributed version is very close to linear.

ZERO++: Harnessing the Power of Zero Appearances to Detect Anomalies in Large-Scale Data Sets

“…Most existing categorical data oriented methods are based on a general assumption that anomalies lie in regions of low frequency (Akoglu et al, 2012;Ghoting, Otey, & Parthasarathy, 2004;He et al, 2005;Koufakou, Ortiz, Georgiopoulos, Anagnostopoulos, & Reynolds, 2007;Koufakou & Georgiopoulos, 2010;Smets & Vreeken, 2011;He, Deng, Xu, & Huang, 2006). Typical examples are frequent patterns based methods FPOF (He et al, 2005) and infrequent patterns based methods LOADED (Ghoting et al, 2004).…”

“…Some computationally expensive algorithms (Ghoting et al, 2004;Koufakou & Georgiopoulos, 2010) require to use parallelism in order to reduce their runtime. But parallelism does not reduce the time complexity of the base algorithm.…”

Outlier Detection and Correction for the Deviations of Tooth Profiles of Gears

“…The lower side of the grey area is determined based on the 2-segment line (line 2). Line 2 is expressed as (11).…”

“…Gear measuring center is a kind of efficient precision measuring instrument, but outliers easily cause sudden measurement environment changes in the measurement of tooth profiles, such as the effect of mechanical vibration on the measuring head and the impact of the sharp current on the data acquisition card [6][7][8]. The outliers lead to the abnormal data and decrease the measurement precision of tooth profiles, so the outliers must be removed as soon as they are detected [9][10][11].…”

Efficient Distributed Decision Trees for Robust Regression

“…For regression problems, it can be very difficult to spot noise or outliers in the data without careful investigation and even harder in multivariate data sets with both categorical and numerical features [13]. Overfitting avoidance, known as the node pruning in the context of regression trees, is a general way to allow robustness for unseen data by penalizing the tree for being too complex.…”