Real-time High Performance Anomaly Detection over Data Streams

Jankov, Dimitrije; Sikdar, Sourav; Mukherjee, Rajarshi; Teymourian, Kia; Jermaine, Chris

doi:10.1145/3093742.3095102

Cited by 18 publications

(18 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In their results that say that their algorithm has good usability and high efficiency. Jankov et al (2017) presents the implementation of a real-time anomaly detection system over data streams. They implement their algorithm in Java and C++ and in this paper, they provide technical details about the data processing pipelines.…”

Section: Related Workmentioning

confidence: 99%

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

Hasani¹

2020

Journal of Computer Science

View full text Add to dashboard Cite

Today's most of the data are streaming time-series data, where is very important anomaly detection over this data because gives significant information of possible critical situations. Detecting anomalies in big streaming data is yet difficult task because we have to process them in real time, even before they are stored and instantly alarm on potential threats. For real time streaming data is important the algorithm used for anomaly detection to be robust with low processing time, eventually at the cost of the accuracy. The aim of this paper is to measure the performance of such algorithms and them to compare with our previously proposed algorithm HW-GA with other existing methods as ARIMA, Moving Average and Holt Winters. The algorithms are implemented in R system and tested on the three Numenta datasets, with known anomalies and own e-dnevnik dataset with unknown anomalies. Evaluation is done by comparing achieved results (the algorithm execution time and CPU usage). As a result of this research we may say that our algorithm HW-GA outperforms others algorithm that we have compared by showing less CPU usage and execution time. Our continues interest is to monitor the streaming log data that are generating in the national educational network (e-dnevnik) that acquires a massive number of online queries and to detect anomalies in order to scale up performance, prevent network downs, alarm on possible attacks and similar.

show abstract

Section: Related Workmentioning

confidence: 99%

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

Hasani¹

2020

Journal of Computer Science

View full text Add to dashboard Cite

show abstract

“…Mairal et al [2] study modeling data vectors as sparse linear combinations of basic elements generating a generic dictionary and then adapt it to specific data. Jankov et al [3] present an implementation of a real-time anomaly detection system over data streams and report experimental results and performance tuning strategies. Vlachos et al [4] formulate the problem of estimating lower/upper distance bounds as an optimization problem and establish the properties of optimal solutions to develop an algorithm which obtains an exact solution to the problem.…”

Section: Literature Reviewmentioning

confidence: 99%

An Adaptive Lightweight Security Framework Suited for IoT

Domb¹

2018

Internet of Things - Technology, Applications and Standardization

View full text Add to dashboard Cite

Standard security systems are widely implemented in the industry. These systems consume considerable computational resources. Devices in the Internet of Things [IoT] are very limited with processing capacity, memory and storage. Therefore, existing security systems are not applicable for IoT. To cope with it, we propose downsizing of existing security processes. In this chapter, we describe three areas, where we reduce the required storage space and processing power. The first is the classification process required for ongoing anomaly detection, whereby values accepted or generated by a sensor are classified as valid or abnormal. We collect historic data and analyze it using machine learning techniques to draw a contour, where all streaming values are expected to fall within the contour space. Hence, the detailed collected data from the sensors are no longer required for real-time anomaly detection. The second area involves the implementation of the Random Forest algorithm to apply distributed and parallel processing for anomaly discovery. The third area is downsizing cryptography calculations, to fit IoT limitations without compromising security. For each area, we present experimental results supporting our approach and implementation.

show abstract

“…There are works oriented specifically to the processing techniques over data streams, such as online outlier detection [20], the join processing [21], etc. These works are based on raw data (or plain tuples), however the PAbMM manage CINCAMI/MIS streams.…”

Section: Related Workmentioning

confidence: 99%

Strategy for the data monetization in tune with the data stream processing

Diván

2017

2017 6th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)

View full text Add to dashboard Cite

In this invited talk, a strategy for the data monetization in tune with the data stream processing is shown. The current organizations have as challenge not only the processing of the big data repositories, but also the data monitoring in real-time for guiding their commercial strategies. When an organization is closer of their customers, it has better knowledge of them and better results will obtain related to their action lines. In this sense, the real time data monitoring is a key asset in the now economy and to know how capitalize them could represent a competitive advantage for any company. In this proposal, the Measurement and Evaluation (M&E) projects oriented to data monetization are organized in terms of the C-INCAMI (Context-Information Need, Concept Model, Attribute, Metric and Indicator) framework for fostering the repeatability, comparability, consistency and extensibility of the measurement process. Therefore, a brief introduction to the M&E framework is done and the Processing Architecture based on Measurement Metadata (PAbMM) is introduced. The PAbMM allows incorporating a detective and predictive behavior based on the data and metadata coming in real time. The interchanging of data between the data sources and PAbMM is supported by the CINCAMI/MIS (Measurement Interchange Schema) which allows keeps the data traceability and collecting a set of complementary data (e.g. geographic data) associated with the entity under analysis and its context. In this sense, the CINCAMI/MIS 2 is summarized and the associated data monetization strategy is detailed. Finally, an application case for exposing the potential associated with the monetization strategy related to PAbMM is shown.

show abstract

Real-time High Performance Anomaly Detection over Data Streams

Cited by 18 publications

References 6 publications

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

Anomaly Detection Algorithms for Streaming Data: Performance Comparison

An Adaptive Lightweight Security Framework Suited for IoT

Strategy for the data monetization in tune with the data stream processing

Contact Info

Product

Resources

About