Detecting Anomaly Teletraffic Using Stochastic Self-Similarity Based on Hadoop

Lee, Jong-Suk R.; Ye, Sang-Kug; Jeong, Hang-Geun

doi:10.1109/nbis.2013.43

Cited by 10 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, there exists various approaches for anomaly detection that considers nonlinear correlation among attributes, but the methods are not scalable to big data. Lee et al proposed an approach by leveraging the capabilities of Hadoop, which is an open‐source distributed framework to detect network anomalies. As network data are very large, so this approach typically addresses concerns related to outlier detection in big data.…”

Section: Related Workmentioning

confidence: 99%

An efficient approach for outlier detection in big sensor data of health care

Saneja

Rani

2017

Int J Communication

View full text Add to dashboard Cite

Summary In recent years, wireless sensor networks are pervasive and are generating tons of data every second. Performing outlier detection to detect faulty sensors from such a large amount of data becomes a challenging task. Most of the existing techniques for outlier detection in wireless sensor networks concentrate only on contents of the data source without considering correlation among different data attributes. Moreover, these methods are not scalable to big data. To address these 2 limitations, this paper proposes an outlier detection approach based on correlation and dynamic SMO (sequential minimal optimization) regression that is scalable to big data. Initially, correlation is used to find out strongly correlated attributes and then the point anomalous nodes are detected using dynamic SMO regression. For fast processing of big data, Hadoop MapReduce framework is used. The experimental analysis demonstrates that the proposed approach efficiently detects the point and contextual anomalies and reduces the number of false alarms. For experiments, real data of sensors used in body sensor networks are taken from Physionet database.

show abstract

Section: Related Workmentioning

confidence: 99%

An efficient approach for outlier detection in big sensor data of health care

Saneja

Rani

2017

Int J Communication

View full text Add to dashboard Cite

show abstract

“…To reduce the effect of the computational complexity of these algorithms, Lee et al [22] have proposed work to detect anomalies by leveraging Hadoop. Hadoop is an opensource software framework that supports applications to run on distributed machines.…”

Section: Processing Pipelinementioning

confidence: 99%

A data-driven approach for multivariate contextualized anomaly detection: Industry use case

Stojanović

Dinic

Stojanović

2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-Anomaly detection is the process of discovering some anomalous behaviour in the real-time operation of a system. It is a difficult task, since in a general case (multivariate anomaly detection, an anomaly can be related to the behaviour of several parameters which are not necessarily behaving anomalously per se, but their (complex) relation is anomalous (not usual/normal). This implies the need for a very efficient modeling of the normal behavior in order to know what should be treated as anomalous/outlier/unusual. Consequently, classical model-driven approaches, due to their focusing on the selected parameters for creating models, are not able to model the behavior if the whole system. This is why data-driven approaches for anomaly detection are getting ever more important for the industry use cases where hundreds (thousands) of parameter should be taken into account. However, current approaches are usually focused on the univariate anomaly detection (or some variations of it), so without going into observing the entire space of relations (computation very difficult). In this paper we present a novel approach for the multivariate anomaly detection that is based on modeling and managing the streams of variations in a multidimensional space. The main advantage of this approach is the possibility to observe the relations between variations of a large set of parameters and create clusters of "normal/usual" variations. In order to ensure scaling, which is one of the most challenging requirements, the approach is based on the usage of the big data technologies for realizing data analytics tasks/calculations. The approach is realized as a part of D2Lab (Data Diagnostics Laboratory) framework and has been applied in several industrial use cases. In this paper we present a very interesting usage for the anomaly detection in the process of functional testing of home appliances devices (in particular case refrigerators) after manufacturing/assembling process. It has been done for a bug vendor (Whirlpool), who expects huge saving in testing and improved customer satisfaction from this approach.

show abstract

“…One issue that the authors present is the difficulty in setting one of the algorithm's parameters. To reduce the effect of the computational complexity of these algorithms, Lee et al [17] have proposed work to detect anomalies by leveraging Hadoop.…”

Section: Profilementioning

confidence: 99%

Contextual anomaly detection framework for big sensor data

Hayes

Capretz

2015

Journal of Big Data

View full text Add to dashboard Cite

The ability to detect and process anomalies for Big Data in real-time is a difficult task. The volume and velocity of the data within many systems makes it difficult for typical algorithms to scale and retain their real-time characteristics. The pervasiveness of data combined with the problem that many existing algorithms only consider the content of the data source; e.g. a sensor reading itself without concern for its context, leaves room for potential improvement. The proposed work defines a contextual anomaly detection framework. It is composed of two distinct steps: content detection and context detection. The content detector is used to determine anomalies in real-time, while possibly, and likely, identifying false positives. The context detector is used to prune the output of the content detector, identifying those anomalies which are considered both content and contextually anomalous. The context detector utilizes the concept of profiles, which are groups of similarly grouped data points generated by a multivariate clustering algorithm. The research has been evaluated against two real-world sensor datasets provided by a local company in Brampton, Canada. Additionally, the framework has been evaluated against the open-source Dodgers dataset, available at the UCI machine learning repository, and against the R statistical toolbox.

show abstract

Detecting Anomaly Teletraffic Using Stochastic Self-Similarity Based on Hadoop

Cited by 10 publications

References 15 publications

An efficient approach for outlier detection in big sensor data of health care

An efficient approach for outlier detection in big sensor data of health care

A data-driven approach for multivariate contextualized anomaly detection: Industry use case

Contextual anomaly detection framework for big sensor data

Contact Info

Product

Resources

About