An efficient algorithm for distributed density-based outlier detection on big data

Bai, Mei; Wang, Xite; Xin, Junchang; Wang, Guoren

doi:10.1016/j.neucom.2015.05.135

Cited by 76 publications

(26 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Traditional methods for detecting anomalies do not show good results in the big data which determined features such as large volumes, variations, high speeds [5][6][7][8].…”

Section: Review Of Computer Engineering Researchmentioning

confidence: 99%

Classification Ensemble Based Anomaly Detection in Network Traffic

Alıguliyev

Hajirahimova

2019

Review of Computer Engineering Research

View full text Add to dashboard Cite

Recently, the expansion of information technologies and the exponential increase of the digital data have deepened more the security and confidentiality issues in computer networks. In the Big Data era information security has become the main direction of scientific research and Big Data analytics is considered being the main tool in the solution of information security issue. Anomaly detection is one of the main issues in data analysis and used widely for detecting network threats. The potential sources of outliers can be noise and errors, events, and malicious attacks on the network. In this work, a short review of network anomaly detection methods is given, is looked at related works. In the article, a more exact and simple multi-classifier model is proposed for anomaly detection in network traffic based on Big Data. Experiments have been performed on the NSL-KDD data set by using the Weka. The offered model has shown decent results in terms of anomaly detection accuracy. Contribution/Originality: This study proposed multi-classifier model for increasing anomaly detection accuracy in network traffic. The model consists of the J48, LogitBoost, IBk, AdaBoost, RandomTree classifiers. This work performed a comparative analysis of used classifiers and their combination to see which one will give the best result In study classifiers and their combination have been implemented on NSL-KDD open source dataset using WEKA tool. The results show that the ensemble classifiers provide the better result than using these classifiers individually. The computer network traffic analysis with employment of our model can help network engineers and administrators to create a more reliable network, avoid possible discharges and take precautionary measures.

show abstract

“…Traditional methods for detecting anomalies do not show good results in the big data which determined features such as large volumes, variations, high speeds [5][6][7][8].…”

Section: Review Of Computer Engineering Researchmentioning

confidence: 99%

Classification Ensemble Based Anomaly Detection in Network Traffic

Alıguliyev

Hajirahimova

2019

Review of Computer Engineering Research

View full text Add to dashboard Cite

show abstract

“…Bai et.al [1] suggests the security mechanism to save the local outlier data from the network intrusion. Here used the two different methods to increase the efficiency such as grid-based portioning algorithm and distributed LOF method.…”

Section: Literature Surveymentioning

confidence: 99%

Grid Partitioning For Anomaly Detection (Gpad) In High Density Distributed Environment For Mining Techniques

Viji¹,

Rajkumar²

2019

IJEAT

View full text Add to dashboard Cite

Anomaly detection is the most important task in data mining techniques. This helps to increase the scalability, accuracy and efficiency. During the extraction process, the outsource may damage their original data set and that will be defined as the intrusion. To avoid the intrusion and maintain the anomaly detection in a high densely populated environment is another difficult task. For that purpose, Grid Partitioning for Anomaly Detection (GPAD) has been proposed for high density environment. This technique will detect the outlier using the grid partitioning approach and density based outlier detection scheme. Initially, all the data sets will be split in the grid format. Allocate the equal amount of data points to each grid. Compare the density of each grid to their neighbor grid in a zigzag manner. Based on the response, lesser density grid will be detected as outlier function as well as that grid will be eliminated. This proposed Grid Partitioning for Anomaly Detection (GPAD) has reduced the complexity and increases the accuracy and these will be proven in simulation part.

show abstract

“…The paper [24] is devoted to the problem of outliers detection in distributed environment based on the determination of the outlier density. In particular, in the data set, for each tuple p , its local outlier factor (LOF) is calculated, which represents the degree of the data set as an outlier.…”

Section: Related Workmentioning

confidence: 99%

An Anomaly Detection Based on Optimization

Alguliyev¹,

Alıguliyev²,

İmamverdiyev³

et al. 2017

IJISA

View full text Add to dashboard Cite

Abstract-At present, an anomaly detection is one of the important problems in many fields. The rapid growth of data volumes requires the availability of a tool for data processing and analysis of a wide variety of data types. The methods for anomaly detection are designed to detect object's deviations from normal behavior. However, it is difficult to select one tool for all types of anomalies due to the increasing computational complexity and the nature of the data. In this paper, an improved optimization approach for a previously known number of clusters, where a weight is assigned to each data point, is proposed. The aim of this article is to show that weighting of each data point improves the clustering solution. The experimental results on three datasets show that the proposed algorithm detects anomalies more accurately. It was compared to the k-means algorithm. The quality of the clustering result was estimated using clustering evaluation metrics. This research shows that the proposed method works better than k-means on the Australia (credit card applications) dataset according to the Purity, Mirkin and F-measure metrics, and on the heart diseases dataset according to F-measure and variation of information metric.

show abstract

An efficient algorithm for distributed density-based outlier detection on big data

Cited by 76 publications

References 18 publications

Classification Ensemble Based Anomaly Detection in Network Traffic

Classification Ensemble Based Anomaly Detection in Network Traffic

Grid Partitioning For Anomaly Detection (Gpad) In High Density Distributed Environment For Mining Techniques

An Anomaly Detection Based on Optimization

Contact Info

Product

Resources

About