Top-k/w publish/subscribe: A publish/subscribe model for continuous top-k processing over data streams

Abstract

Continuous processing of top-k queries over data streams is a promising technique for alleviating the information overload problem as it distinguishes relevant from irrelevant data stream objects with respect to a given scoring function over time. Thus it enables filtering of irrelevant data objects and delivery of top-k objects relevant to user interests in real-time. We propose a solution for distributed continuous top-k processing based on the publish/subscribe communication paradigm-top-k publish/subscribe over sliding windows (top-k/w publish/subscribe). It identifies k best-ranked objects with respect to a given scoring function over a sliding window of size w, and extends the publish/subscribe communication paradigm by continuous top-k processing algorithms coming from the field of data stream processing.In this paper, we introduce, analyze and evaluate the essential building blocks of distributed top-k/w publish/subscribe systems: First, we present a formal top-k/w publish/subscribe model and compare it to the prevailing Boolean publish/subscribe model. Next, we outline the top-k/w processing tasks performed by publish/subscribe nodes and investigate the properties of supported scoring functions. Furthermore, we explore potential routing strategies for distributed top-k/w publish/subscribe systems. Finally, we experimentally evaluate model properties and provide a comparative study investigating traffic requirements of potential routing strategies.

Energy-aware and quality-driven sensor management for green mobile crowd sensing

“…One possible option is adopting a top-k/w approach from Pripužić et al (2014Pripužić et al ( , 2015 to continuously find the top-k elements (sensors) in a fixed geographic-window, i.e., the window corresponding to a specific cell of interest. The value of k corresponds to the minimal number of sensors that is needed within the cell of interest and in a given time interval to obtain the required sensor readings, while sensors are ranked based on a specified valuation function, as previously discussed.…”

Threshold-Based Distributed Continuous Top-k Query Processing for Minimizing Communication Overhead

“…[16] addresses multiple continuous top-k queries over data streams in centralized databases by aiming at reducing CPU cost. In [17], the authors proposed a real-time publish/subscribe model to perform continuous top-k query processing, while our problem is to reduce the communication cost of monitoring top-k answers on high-update horizontally partitioned databases. The works above are different from ours because they assume continuous queries on data streams over sliding windows or time-frame models.…”

“…Therefore, BS need to construct and disseminate the subscriptions of those new dominating queries. After that BS has to inform the nodes which store this query's subscription to invalidate it from S L i as shown in Algorithm 1 (lines [16][17][18][19][20].…”

A high throughput processing engine for taxi-generated data streams

“…Since a large number of routes can be active in the engine and the output is generated only when a change occurs in the top 10 ranking, it is inefficient to keep record about all routes. We reuse the top-k/w approach which in addition to top-k routes also maintains that have potential to become top-10 routes [18]. We used an array to store the best ranked routes, where the index value indicates route's position.…”