Communication-efficient distributed monitoring of thresholded counts

Abstract

Monitoring is an issue of primary concern in current and next generation networked systems. For example, the objective of sensor networks is to monitor their surroundings for a variety of different applications like atmospheric conditions, wildlife behavior, and troop movements among others. Similarly, monitoring in data networks is critical not only for accounting and management, but also for detecting anomalies and attacks. Such monitoring applications are inherently continuous and distributed, and must be designed to minimize the communication overhead that they introduce. In this context we introduce and study a fundamental class of problems called "thresholded counts" where we must return the aggregate frequency count of an event that is continuously monitored by distributed nodes with a user-specified accuracy whenever the actual count exceeds a given threshold value.In this paper we propose to address the problem of thresholded counts by setting local thresholds at each monitoring node and initiating communication only when the locally observed data exceeds these local thresholds. We explore algorithms in two categories: static thresholds and adaptive thresholds. In the static case, we consider thresholds based on a linear combination of two alternate strategies, and show that there exists an optimal blend of the two strategies that results in minimum communication overhead. We further show that this optimal blend can be found using a steepest descent search. In the adaptive case, we propose algorithms that adjust the local thresholds based on the observed distributions of updated information in the distributed monitoring system. We use extensive simulations not only to verify the accuracy of our algorithms and validate our theoretical results, but also to evaluate the performance of the two approaches. We find that both approaches yield significant savings over the naive approach of performing processing at a centralized location.

Decentralized detection of global threshold crossings using aggregation trees

“…Next, it checks whether a node needs to switch state from active to passive or vice versa. If the local threshold is 0 or if the partial aggregate is larger than the upper hysteresis threshold (or smaller than the lower threshold in negative mode), then the node, if not already active, switches to active state and resets the threshold of its children to 0 (lines [2][3][4][5]. If the node is active and the aggregate is below the lower hysteresis threshold (or above the upper hysteresis threshold in negative mode), then it switches to passive state and sets the threshold of its children (see section 3.3).…”

Prediction-based geometric monitoring over distributed data streams

“…In related work regarding distributed trigger monitoring, [13] provides a framework for monitoring thresholded counts over distributed data streams, while [12] designs techniques that decompose the problem of detecting when the sum of a distributed set of variables exceeds a given threshold. Based on [12] anomaly detection techniques are studied in [10] and [11].…”

“…An important query type that is of the essence in the aforementioned fields regards the monitoring of a trigger condition defined upon the range of values a function of interest receives [18,20,21,10,12,13,11]. For instance, in order to perform spam detection on a number of dispersed mail servers, algorithms base their decisions on whether the value of the information gain function globally exceeds a given threshold [20].…”

Monitoring infrastructure for converged networks and services

“…Examples include congestion detection, SLA violation detection, and distributed denial of service (DDoS) attack detection. References [1,9,10,13] discuss algorithms to implement distributed constraint monitoring efficiently.…”