Statistical hierarchical clustering algorithm for outlier detection in evolving data streams

Abstract

Anomaly detection is a hard data analysis process that requires constant creation and improvement of data analysis algorithms. Using traditional clustering algorithms to analyse data streams is impossible due to processing power and memory issues. To solve this, the traditional clustering algorithm complexity needed to be reduced, which led to the creation of sequential clustering algorithms. The usual approach is two-phase clustering, which uses online phase to relax data details and complexity, and offline phase to cluster concepts created in the online phase. Detecting anomalies in a data stream is usually solved in the online phase, as it requires unreduced data. Contrarily, producing good macro-clustering is done in the offline phase, which is the reason why two-phase clustering algorithms have difficulty being equally good in anomaly detection and macro-clustering. In this paper, we propose a statistical hierarchical clustering algorithm equally suitable for both detecting anomalies and macro-clustering. The proposed algorithm is single-phased and uses statistical inference on the input data stream, resulting in statistical distributions that are constantly updated. This makes the classification adaptable, allowing agglomeration of outliers into clusters, tracking population evolution, and to be used without knowing the expected number of clusters and outliers. The proposed algorithm was tested against typical clustering algorithms, including two-phase algorithms suitable for data stream analysis. A number of typical test cases were selected, to show the universality and qualities of the proposed clustering algorithm.

A High-Dimensional and Small-Sample Submersible Fault Detection Method Based on Feature Selection and Data Augmentation

“…In this work, agglomerative hierarchical clustering algorithm [ 37 ] is used as the feature grouping method. As shown in Figure 4 , the agglomerative hierarchical clustering method initially treats each feature as a cluster, and then combines the two most similar clusters into a new larger cluster step by step.…”

Hydrologic similarity based on width function and hypsometry: An unsupervised learning approach

“…Intra-cluster uniformity can be further improved by lowering this threshold. While rigorous methods for removal of outliers exist in the literature (Almeida et al, 2007;Fan et al, 2013;Krleža et al, 2021), we employed this basic outlier detection algorithm as a proof of concept, one that is easy to understand and can be readily applied.…”

Hierarchical clustering framework for facility location selection with practical constraints

“…Note that the locations in remote regions can potentially form small clusters or even become isolated from other locations in hierarchical clustering due to the bottom-up nature [29]. In real business, outliers are not allowed to appear.…”