Efficient anomaly detection in dynamic, attributed graphs: Emerging phenomena and big data

Dinic

2017 IEEE International Conference on Big Data (Big Data)

2017

Abstract-Anomaly detection is the process of discovering some anomalous behaviour in the real-time operation of a system. It is a difficult task, since in a general case (multivariate anomaly detection, an anomaly can be related to the behaviour of several parameters which are not necessarily behaving anomalously per se, but their (complex) relation is anomalous (not usual/normal). This implies the need for a very efficient modeling of the normal behavior in order to know what should be treated as anomalous/outlier/unusual. Consequently, classical model-driven approaches, due to their focusing on the selected parameters for creating models, are not able to model the behavior if the whole system. This is why data-driven approaches for anomaly detection are getting ever more important for the industry use cases where hundreds (thousands) of parameter should be taken into account. However, current approaches are usually focused on the univariate anomaly detection (or some variations of it), so without going into observing the entire space of relations (computation very difficult). In this paper we present a novel approach for the multivariate anomaly detection that is based on modeling and managing the streams of variations in a multidimensional space. The main advantage of this approach is the possibility to observe the relations between variations of a large set of parameters and create clusters of "normal/usual" variations. In order to ensure scaling, which is one of the most challenging requirements, the approach is based on the usage of the big data technologies for realizing data analytics tasks/calculations. The approach is realized as a part of D2Lab (Data Diagnostics Laboratory) framework and has been applied in several industrial use cases. In this paper we present a very interesting usage for the anomaly detection in the process of functional testing of home appliances devices (in particular case refrigerators) after manufacturing/assembling process. It has been done for a bug vendor (Whirlpool), who expects huge saving in testing and improved customer satisfaction from this approach.

“…Miller et al [20] discuss anomaly detection in the domain of attributed graphs. Their work allows for contextual data to be included within a graph structure.…”

Section: Processing Pipelinementioning

confidence: 99%

A data-driven approach for multivariate contextualized anomaly detection: Industry use case

Dinic

2017 IEEE International Conference on Big Data (Big Data)

2017

“…Miller et al [15] discuss anomaly detection in the domain of attributed graphs. Their work allows for contextual data to be included within a graph structure.…”

Section: Profilementioning

confidence: 99%

“…One interesting result is that considering additional metadata forced the algorithm to explore parts of the graph that were previously less emphasized. A drawback of Miller et al 's [15] work is that their full algorithm is difficult for use in real-time analytics. To compensate, they provide an estimation of their algorithm for use in real-time analytics, however the estimation is not explored in detail and so it is difficult to determine its usefulness in the real-time detection domain.…”

Section: Profilementioning

confidence: 99%

Contextual anomaly detection framework for big sensor data

Hayes

Capretz

2015

Journal of Big Data

The ability to detect and process anomalies for Big Data in real-time is a difficult task. The volume and velocity of the data within many systems makes it difficult for typical algorithms to scale and retain their real-time characteristics. The pervasiveness of data combined with the problem that many existing algorithms only consider the content of the data source; e.g. a sensor reading itself without concern for its context, leaves room for potential improvement. The proposed work defines a contextual anomaly detection framework. It is composed of two distinct steps: content detection and context detection. The content detector is used to determine anomalies in real-time, while possibly, and likely, identifying false positives. The context detector is used to prune the output of the content detector, identifying those anomalies which are considered both content and contextually anomalous. The context detector utilizes the concept of profiles, which are groups of similarly grouped data points generated by a multivariate clustering algorithm. The research has been evaluated against two real-world sensor datasets provided by a local company in Brampton, Canada. Additionally, the framework has been evaluated against the open-source Dodgers dataset, available at the UCI machine learning repository, and against the R statistical toolbox.

2014 IEEE International Congress on Big Data

“…Miller et al [10] discuss anomaly detection in the domain of attributed graphs. Their work allows for contextual data to be included within a graph structure.…”

Section: Related Workmentioning

confidence: 99%

“…One interesting result is that considering additional metadata forced the algorithm to explore parts of the graph that were previously less emphasized. A drawback of Miller et al's [10] work is that their full algorithm is difficult for use in real-time analytics. To compensate, they provide an estimation of their algorithm for use in real-time analytics, however the estimation is not explored in detail and so it is difficult to determine its usefulness in the real-time detection domain.…”

Section: Related Workmentioning

confidence: 99%

Contextual Anomaly Detection in Big Sensor Data

Hayes

Capretz

2014

Abstract-Performing predictive modelling, such as anomaly detection, in Big Data is a difficult task. This problem is compounded as more and more sources of Big Data are generated from environmental sensors, logging applications, and the Internet of Things. Further, most current techniques for anomaly detection only consider the content of the data source, i.e. the data itself, without concern for the context of the data. As data becomes more complex it is increasingly important to bias anomaly detection techniques for the context, whether it is spatial, temporal, or semantic. The work proposed in this paper outlines a contextual anomaly detection technique for use in streaming sensor networks. The technique uses a well-defined content anomaly detection algorithm for real-time point anomaly detection. Additionally, we present a post-processing contextaware anomaly detection algorithm based on sensor profiles, which are groups of contextually similar sensors generated by a multivariate clustering algorithm. Our proposed research has been implemented and evaluated with real-world data provided by Powersmiths, located in Brampton, Ontario, Canada.