Network-Aware Query Processing for Stream-based Applications

Abstract

This paper investigates the benefits of network awareness when processing queries in widelydistributed environments such as the Internet. We present algorithms that leverage knowledge of network characteristics (e.g., topology, bandwidth, etc.) when deciding on the network locations where the query operators are executed. Using a detailed emulation study based on realistic network models, we analyse and experimentally evaluate the proposed approaches for distributed stream processing. Our results quantify the significant benefits of the network-aware approaches and reveal the fundamental trade-off between bandwidth efficiency and result latency that arises in networked query processing.

Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System

“…On the other hand, [3,8] focused on minimizing communication cost but ignores load balancing. [10] studied the static operator placement in a hierarchical stream acquisition architecture, which is much different from our system architecture.…”

“…More specifically, the heuristics (1) balance the load among all the processing nodes; (2) restrict the number of nodes that the operators of a query can be distributed to; (3) and minimize the total communication cost under conditions (1) and (2).…”

Biologically-Inspired Distributed Middleware Management for Stream Processing Systems

“…Decentralized algorithms have been proposed to minimize network usage and dynamically adjust placement in response to network changes during runtime [19,20]. These algorithms focus on minimizing communication cost, and do not explicitly have a load-balancing strategy.…”

Hot-spot prediction and alleviation in distributed stream processing applications

“…Examples include analyzing the input provided by website visitors to provide relevant advertising, monitoring network traffic to detect intrusions or update router configuration, customizing news feeds to user interests, or processing financial trading data for recommendations or alerts. This type of applications has given rise to a new class of systems, called Distributed Stream Processing Systems (DSPSs) [1,8,10,13,14]. In DSPSs, reusable components, located on geographically distributed nodes, process continuous data streams in real-time.…”

“…To drive migration decisions in a decentralized manner we build a load monitoring architecture on top of a Distributed Hash Table (DHT) [16]. iii) We have implemented our techniques in Synergy [14], our distributed stream processing middleware 1 . To validate our approach we have deployed our middleware on the PlanetLab [5] wide-area network testbed and we have run experiments of a real network monitoring application [20] operating on traces of real TCP traffic [22].…”