A fast and noise resilient cluster-based anomaly detection

Bigdeli, Elnaz; Mohammadi, Mahdi; Raahemi, Bijan; Matwin, Stan

doi:10.1007/s10044-015-0484-0

Cited by 19 publications

(9 citation statements)

References 34 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare our proposed algorithm NoiseCleaner with five noisy values identification methods, namely HARF-80 and HARF-70 from NOISERANK [16], HCleaner [21], CPAD [24], and CAIRAD [13]. In NOISERANK, there are several ensemble methods.…”

Section: Resultsmentioning

confidence: 99%

“…Identification of noisy data is an important data preprocessing task for improving data quality. Many noise detection algorithms have been proposed for various applications [2,5,13,16,17,21,23,24]. Among them, HCleaner [21], NOISERANK [16], Polishing method [17] and Error Detection and Impact-sensitive instance Ranking (EDIR) [23] are some wellknown noisy value detection algorithms.…”

Section: Related Workmentioning

confidence: 99%

“…Bigdeli et al [24] proposed the Collective Probabilistic Anomaly Detection (CPAD) method to increase the accuracy of the anomaly detection by focusing on arbitrary shape…”

Section: Accepted Manuscriptmentioning

confidence: 99%

See 2 more Smart Citations

Noisy values detection and correction of traffic accident data

Deb

Liew

2019

Information Sciences

View full text Add to dashboard Cite

Death, injury, and disability from road traffic crashes continue to be a major global public health problem. Therefore, methods to reduce accident severity are of significant interest to traffic agencies and the public at large. Noisy data in the traffic accident dataset obscure the discovery of important factors and mislead conclusions. Identifying and correcting noisy values is an important goal of data cleansing and preprocessing. This paper proposes a new algorithm called NoiseCleaner to identify and correct noisy categorical attributes values in large traffic accident datasets. We evaluate our algorithm using four publicly available traffic accident datasets from Australia and United States, namely, two road crash datasets from the Queensland Government data depository (data.qld.gov.au) and two datasets from the New York's open data portal (data.ny.gov). We compare our technique with several existing stateof-the-art methods and show that our algorithm performs significantly better than the existing algorithms.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Noisy values detection and correction of traffic accident data

Deb

Liew

2019

Information Sciences

View full text Add to dashboard Cite

show abstract

“…• A fast and noise resilient cluster-based anomaly detection [E.Bigdeli et al, 2015a] • Incremental Cluster Updating Using Gaussian Mixture Model [E.Bigdeli et al, 2015c] • A Fast Noise Resilient Anomaly Detection using GMM-based Collective…”

Section: Published Papersmentioning

confidence: 99%

Incremental anomaly detection using two-layer cluster-based structure

Bigdeli

Mohammadi

Raahemi

et al. 2018

Information Sciences

Self Cite

View full text Add to dashboard Cite

Anomaly detection algorithms face several challenges, including processing speed and dealing with noise in data. In this thesis, a two-layer clusterbased anomaly detection structure is presented which is fast, noise-resilient and incremental. In this structure, each normal pattern is considered as a cluster, and each cluster is represented using a Gaussian Mixture Model (GMM). Then, new instances are presented to the GMM to be labeled as normal or abnormal.The proposed structure comprises three main steps. In the first step, the data are clustered. The second step is to represent each cluster in a way that enables the model to classify new instances. The Summarization based on Gaussian Mixture Model (SGMM) proposed in this thesis represents each cluster as a GMM.In the third step, a two-layer structure efficiently updates clusters using In most real-time anomaly detection applications, incoming instances are often similar to previous ones. In these cases, there is no need to update clusters based on duplicates, since they have already been modeled in the cluster distribution. The two-layer structure is responsible for identifying redundant instances. In this structure, redundant instance are ignored, and the remaining new instances are used to update clusters. Ignoring redundant instances, which are typically in the majority, makes the detection phase fast.Each part of the general structure is validated in this thesis. The experiments include, detection rates, clustering goodness, time, memory usage and the complexity of the algorithms. The accuracy of the clustering and summarization of clusters using GMMs is evaluated, and compared to that of other methods. Using Davies-Bouldin (DB) and Dunn indexes, the distances for original and regenerated clusters using GMMs is almost zero with SGMM method while this value for ABACUS is around 0.01. Moreover, the results show that the SGMM algorithm is 3 times faster than ABACUS in running time, using one-third of the memory used by ABACUS.The CPL method, used to label new instances, is found to collectively remove the effect of noise, while increasing the accuracy of labeling new instances. In a noisy environment, the detection rate of the CPL method is 5% higher than other algorithms such as one-class SVM. The false alarm iii rate is decreased by 10% on average. Memory use is 20 times lesser that that of the one-class SVM.The proposed method is found to lower the false alarm rate, which is one of the basic problems for the one-class SVM. Experiments show the false alarm rate is decreased from 5% to 15% among different datasets, while the detection rate is increased from 5% to 10% in different datasets with twolayer structure. The memory usage for the two-layer structure is 20 to 50 times less than that of one-class SVM. One-class SVM uses support vectors in labeling new instances, while the labeling of the two-layer structure depends on the number of GMMs. The experiments show that the two-layer structure is 20 to 50 times faster than the one-class SVM in labelin...

show abstract

“…In this type, various common statistical properties of distributions, including mean, median, and mode are used to identify irregularities in data. The statistical analysis based filters are good at anomaly detections but for highly uncertain data sets, these may generate false information [15]. The noise among users is also considered as an abnormal behavior and generated as false anomalies.…”

Section: Statistical-basedmentioning

confidence: 99%

A Survey on the Security of Pervasive Online Social Networks (POSNs)

Gupta,

Choudhary,

Sharma

2018

Preprint

View full text Add to dashboard Cite

Pervasive Online Social Networks (POSNs) are the extensions of Online Social Networks (OSNs) which facilitate connectivity irrespective of the domain and properties of users. POSNs have been accumulated with the convergence of a plethora of social networking platforms with a motivation of bridging their gap. Over the last decade, OSNs have visually perceived an altogether tremendous amount of advancement in terms of the number of users as well as technology enablers. A single OSN is the property of an organization, which ascertains smooth functioning of its accommodations for providing a quality experience to their users. However, with POSNs, multiple OSNs have coalesced through communities, circles, or only properties, which make service-provisioning tedious and arduous to sustain. Especially, challenges become rigorous when the focus is on the security perspective of cross-platform OSNs, which are an integral part of POSNs. Thus, it is of utmost paramountcy to highlight such a requirement and understand the current situation while discussing the available state-of-the-art. With the modernization of OSNs and convergence towards POSNs, it is compulsory to understand the impact and reach of current solutions for enhancing the security of users as well as associated services. This survey understands this requisite and fixates on different sets of studies presented over the last few years and surveys them for their applicability to POSNs. There is a limited amount of content available for the security of POSNs. However, being an extension to general OSNs, solutions applicable to OSNs are withal included to understand the practicality for POSNs. Moreover, this survey additionally includes content cognate to trust management and anomaly detection in POSNs. In integration, a broad classification is additionally presented for each category with a tabular comparison. At last, certain future challenges, open issues, and research goals are presented, which can be focused by leading or upcoming researchers while emphasizing the security of POSNs.

show abstract

A fast and noise resilient cluster-based anomaly detection

Cited by 19 publications

References 34 publications

Noisy values detection and correction of traffic accident data

Noisy values detection and correction of traffic accident data

Incremental anomaly detection using two-layer cluster-based structure

A Survey on the Security of Pervasive Online Social Networks (POSNs)

Contact Info

Product

Resources

About