A fast and noise resilient cluster-based anomaly detection

Abstract

Clustering, while systematically applied in anomaly detection, has a direct impact on the accuracy of the detection methods. Existing cluster-based anomaly detection methods are mainly based on spherical shape clustering. In this paper, we focus on arbitrary shape clustering methods to increase the accuracy of the anomaly detection. However, since the main drawback of arbitrary shape clustering is its high memory complexity, we propose to summarize clusters first. For this, we design an algorithm, called Summarization based on Gaussian Mixture Model (SGMM), to summarize clusters and represent them as Gaussian Mixture Models (GMMs). After GMMs are constructed, incoming new samples are presented to the GMMs, and their membership values are calculated, based on which the new samples are labeled as ''normal'' or ''anomaly.'' Additionally, to address the issue of noise in the data, instead of labeling samples individually, they are clustered first, and then each cluster is labeled collectively. For this, we present a new approach, called Collective Probabilistic Anomaly Detection (CPAD), in which, the distance of the incoming new samples and the existing SGMMs is calculated, and then the new cluster is labeled the same as of the closest cluster. To measure the distance of two GMM-based clusters, we propose a modified version of the Kullback-Libner measure. We run several experiments to evaluate the performances of the proposed SGMM and CPAD methods and compare them against some of the wellknown algorithms including ABACUS, local outlier factor (LOF), and one-class support vector machine (SVM). The performance of SGMM is compared with ABACUS using Dunn and DB metrics, and the results indicate that the SGMM performs superior in terms of summarizing clusters. Moreover, the proposed CPAD method is compared with the LOF and one-class SVM considering the performance criteria of (a) false alarm rate, (b) detection rate, and (c) memory efficiency. The experimental results show that the CPAD method is noise resilient, memory efficient, and its accuracy is higher than the other methods.

Noisy values detection and correction of traffic accident data

“…We compare our proposed algorithm NoiseCleaner with five noisy values identification methods, namely HARF-80 and HARF-70 from NOISERANK [16], HCleaner [21], CPAD [24], and CAIRAD [13]. In NOISERANK, there are several ensemble methods.…”

Noisy values detection and correction of traffic accident data

“…We compare our proposed algorithm NoiseCleaner with five noisy values identification methods, namely HARF-80 and HARF-70 from NOISERANK [16], HCleaner [21], CPAD [24], and CAIRAD [13]. In NOISERANK, there are several ensemble methods.…”

Noise-Aware Misclassification Attack Detection in Collaborative DNN Inference

“…Noisy samples are unavoidable in real-world applications. They can distort the boundaries between normal and abnormal behavior, making it harder to distinguish benign noise-affected data from truly adversarial samples [25]. As a result, it becomes essential to account for the behavior of such noise when designing reliable adversarial detection methods.…”

One Class Support Subsequence Machine: Abnormal time series subsequences detection using One-Class SVM

“…However, a shortcoming of clustering-based methods is that they consider the clusters to be spherical. (Bigdeli et al, 2017) An important number of dissimilarity-based methods for anomaly detection are based on the k-NN algorithm. Ishimtsev et al (2017) use the average of the distances of the subsequence of interest S to its k nearest neighbors in Ŝ as the anomaly score.…”

Noisy values detection and correction of traffic accident data

“…We compare our proposed algorithm NoiseCleaner with five noisy values identification methods, namely HARF-80 and HARF-70 from NOISERANK [16], HCleaner [21], CPAD [24], and CAIRAD [13]. In NOISERANK, there are several ensemble methods.…”

Noise-Aware Misclassification Attack Detection in Collaborative DNN Inference

“…Noisy samples are unavoidable in real-world applications. They can distort the boundaries between normal and abnormal behavior, making it harder to distinguish benign noise-affected data from truly adversarial samples [25]. As a result, it becomes essential to account for the behavior of such noise when designing reliable adversarial detection methods.…”

One Class Support Subsequence Machine: Abnormal time series subsequences detection using One-Class SVM

“…However, a shortcoming of clustering-based methods is that they consider the clusters to be spherical. (Bigdeli et al, 2017) An important number of dissimilarity-based methods for anomaly detection are based on the k-NN algorithm. Ishimtsev et al (2017) use the average of the distances of the subsequence of interest S to its k nearest neighbors in Ŝ as the anomaly score.…”

Noisy values detection and correction of traffic accident data

“…We compare our proposed algorithm NoiseCleaner with five noisy values identification methods, namely HARF-80 and HARF-70 from NOISERANK [16], HCleaner [21], CPAD [24], and CAIRAD [13]. In NOISERANK, there are several ensemble methods.…”

Noise-Aware Misclassification Attack Detection in Collaborative DNN Inference

“…Noisy samples are unavoidable in real-world applications. They can distort the boundaries between normal and abnormal behavior, making it harder to distinguish benign noise-affected data from truly adversarial samples [25]. As a result, it becomes essential to account for the behavior of such noise when designing reliable adversarial detection methods.…”

One Class Support Subsequence Machine: Abnormal time series subsequences detection using One-Class SVM

“…However, a shortcoming of clustering-based methods is that they consider the clusters to be spherical. (Bigdeli et al, 2017) An important number of dissimilarity-based methods for anomaly detection are based on the k-NN algorithm. Ishimtsev et al (2017) use the average of the distances of the subsequence of interest S to its k nearest neighbors in Ŝ as the anomaly score.…”