Communication-Efficient Online Detection of Network-Wide Anomalies

Huang, Ling; Nguyen, Xuan Long; Garofalakis, Minos; Hellerstein, Joseph M.; Jordan, Michael I.; Joseph, Anthony D.; Taft, Nina

doi:10.1109/infcom.2007.24

Cited by 109 publications

(83 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In several review papers [26][27][28][29][30][31][32] various network anomaly detection methods have been summarized. From aforementioned surveys one can find that the most effective methods of network anomaly detection are Principle Component Analysis [33][34][35], Wavelet analysis [36][37][38], Markovian models [39,40], Clustering [41][42][43], Histograms [44,45], Sketches [46,47], and Entropies [8,15,48].…”

Section: General Overview Of Network Anomaly Techniquesmentioning

confidence: 99%

An Entropy-Based Network Anomaly Detection Method

Bereziński

Jasiul

Szpyrka

2015

Entropy

156

View full text Add to dashboard Cite

Data mining is an interdisciplinary subfield of computer science involving methods at the intersection of artificial intelligence, machine learning and statistics. One of the data mining tasks is anomaly detection which is the analysis of large quantities of data to identify items, events or observations which do not conform to an expected pattern. Anomaly detection is applicable in a variety of domains, e.g., fraud detection, fault detection, system health monitoring but this article focuses on application of anomaly detection in the field of network intrusion detection.The main goal of the article is to prove that an entropy-based approach is suitable to detect modern botnet-like malware based on anomalous patterns in network. This aim is achieved by realization of the following points: (i) preparation of a concept of original entropy-based network anomaly detection method, (ii) implementation of the method, (iii) preparation of original dataset, (iv) evaluation of the method.

show abstract

Section: General Overview Of Network Anomaly Techniquesmentioning

confidence: 99%

An Entropy-Based Network Anomaly Detection Method

Bereziński

Jasiul

Szpyrka

2015

Entropy

156

View full text Add to dashboard Cite

show abstract

“…This generalizes the classic reservoir sampling problem (see, e.g., [2], where the algorithm is attributed to Waterman; see also [3]) to the setting of multiple distributed streams, and has applications to approximate query answering, selectivity estimation, and query planning. For example, in the case of network routers, maintaining a random sample from the union of the streams is valuable for network monitoring tasks involving the detection of global properties [4]. Other problems on distributed stream processing, including the estimation of the number of distinct elements [1], [5] and heavy hitters [6], [7], [8], [9], use random sampling as a primitive (we note, though, that better solutions for the heavy hitters problem in terms of the accuracy parameter may be possible [9] than those provided by random sampling).…”

Section: Introductionmentioning

confidence: 99%

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Chung

Tirthapura

Woodruff

2016

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Abstract-We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Keywords

show abstract

“…Based on [12] anomaly detection techniques are studied in [10] and [11]. The recent work of [6] provides upper and lower communication bounds for approximate monitoring of thresholded F p moments, with p = 0, 1, 2.…”

Section: Related Workmentioning

confidence: 99%

“…An important query type that is of the essence in the aforementioned fields regards the monitoring of a trigger condition defined upon the range of values a function of interest receives [18,20,21,10,12,13,11]. For instance, in order to perform spam detection on a number of dispersed mail servers, algorithms base their decisions on whether the value of the information gain function globally exceeds a given threshold [20].…”

Section: Introductionmentioning

confidence: 99%

Prediction-based geometric monitoring over distributed data streams

Giatrakos

Deligiannakis

Garofalakis

et al. 2012

Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Many modern streaming applications, such as online analysis of financial, network, sensor and other forms of data are inherently distributed in nature. An important query type that is the focal point in such application scenarios regards actuation queries, where proper action is dictated based on a trigger condition placed upon the current value that a monitored function receives. Recent work [18,20,21] studies the problem of (non-linear) sophisticated function tracking in a distributed manner. The main concept behind the geometric monitoring approach proposed there, is for each distributed site to perform the function monitoring over an appropriate subset of the input domain. In the current work, we examine whether the distributed monitoring mechanism can become more efficient, in terms of the number of communicated messages, by extending the geometric monitoring framework to utilize prediction models. We initially describe a number of local estimators (predictors) that are useful for the applications that we consider and which have already been shown particularly useful in past work. We then demonstrate the feasibility of incorporating predictors in the geometric monitoring framework and show that prediction-based geometric monitoring in fact generalizes the original geometric monitoring framework. We propose a large variety of different predictionbased monitoring models for the distributed threshold monitoring of complex functions. Our extensive experimentation with a variety of real data sets, functions and parameter settings indicates that our approaches can provide significant communication savings ranging between two times and up to three orders of magnitude, compared to the transmission cost of the original monitoring framework.

show abstract

Communication-Efficient Online Detection of Network-Wide Anomalies

Cited by 109 publications

References 10 publications

An Entropy-Based Network Anomaly Detection Method

An Entropy-Based Network Anomaly Detection Method

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Prediction-based geometric monitoring over distributed data streams

Contact Info

Product

Resources

About