A Novel Density-based Technique for Outlier Detection of High Dimensional Data Utilizing Full Feature Space

Abstract

Recently, anomaly detection has acquired a realistic response from data mining scientists as a graph of its reputation has increased smoothly in various practical domains like product marketing, fraud detection, medical diagnosis, fault detection and so many other fields. High dimensional data subjected to outlier detection poses exceptional challenges for data mining experts and it is because of natural problems of the curse of dimensionality and resemblance of distant and adjoining points. Traditional algorithms and techniques were experimented on full feature space regarding outlier detection. Customary methodologies concentrate largely on low dimensional data and hence show ineffectiveness while discovering anomalies in a data set comprised of a high number of dimensions. It becomes a very difficult and tiresome job to dig out anomalies present in high dimensional data set when all subsets of projections need to be explored. All data points in high dimensional data behave like similar observations because of its intrinsic feature i.e., the distance between observations approaches to zero as the number of dimensions extends towards infinity. This research work proposes a novel technique that explores deviation among all data points and embeds its findings inside well established density-based techniques. This is a state of art technique as it gives a new breadth of research towards resolving inherent problems of high dimensional data where outliers reside within clusters having different densities. A high dimensional dataset from UCI Machine Learning Repository is chosen to test the proposed technique and then its results are compared with that of density-based techniques to evaluate its efficiency.

Detecting Outliers in Data Streams Based on Minimum Rare Pattern Mining and Pattern Matching

“…Many previous OD studies [3,13,28,35,36] have not considered the appearing frequency of each data sample in the detection process, thus, the detected outliers could not fit their two characteristics well. Different with these OD methods, the appearing frequency of patterns is also considered as an important factor that will influence the detection accuracy in pattern matching-based method [8], thus, its detection accuracy is much higher.…”

“…The use of anti-monotonic constraint property can help to reduce extensible patterns, and thus saving the time cost. Many previous OD studies [3,13,28,35,36] have not considered the appearing frequency of each data sample in the detection process, thus, the detected outliers could not fit their two characteristics well. Different with these OD methods, the appearing frequency of patterns is also considered as an important factor that will traditional pattern matching-based OD is very long because of the large amount of RPs can be mined.…”

“…In addition to pattern matching, the statistical analysis technology and machine learning technology are also used in OD. In recent years, some OD algorithms, such as clustering-based [21,24,27], distance-based [3,13], density-based [28,35,36], model-based [5] have also been used in practical applications of specific domains.…”

Graph Convolutional Networks and Attention-Based Outlier Detection

“…The o represents another point o in the matrix, Nk(p) represents the set of k nearest neighbors of p,and lrdk(p) represents the locally reachable density at point p. kdistance(o) denotes the location of the kth distance from o, excluding the point o. d(o,p) means the distance between the point o and the point p. The formula is ( 17): (17) The larger the LOFk(p) value, the more likely it is to be an outlier. Finally, the outlier points can be derived by performing a descending sorting operation on the local outlier factors of each object in matrix Z and comparing the magnitude with a custom threshold.…”

“…Outlier detection has an extensive range of applications in many fields: bank fraud [2]- [4], video surveillance [5]- [8], network anomalies [9]- [11], finding new celestial objects [12], [13], etc. The available outlier detection algorithms can be broadly classified into: distance-based algorithms [14]- [17], density-based algorithms [18]- [20], clustering-based algorithms [21], [22], statistical methods [23], integration-based methods [24], numerous neural network-based algorithms [25] and graph-based algorithms [26] etc. Most graph algorithms only work on graph-structured data, and the same goes for GCN [27].…”