Near-optimal algorithms for shared filter evaluation in data stream systems

Abstract

We consider the problem of evaluating multiple overlapping queries defined on data streams, where each query is a conjunction of multiple filters and each filter may be shared across multiple queries. Efficient support for overlapping queries is a critical issue in the emerging data stream systems, and this is particularly the case when filters are expensive in terms of their computational complexity and processing time. This problem generalizes other well-known problems such as pipelined filter ordering and set cover, and is not only NP-Hard but also hard to approximate within a factor of o(log n) from the optimum, where n is the number of queries. In this paper, we present two near-optimal approximation algorithms with provably-good performance guarantees for the evaluation of overlapping queries. We present an edge-coverage based Greedy algorithm which achieves an approximation ratio of (1 + log(n) + log(α)), where n is the number of queries and α is the average number of filters in a query. We also present a randomized, fast and easily parallelizable Harmonic algorithm which achieves an approximation ratio of 2β, where β is the maximum number of filters in a query. We have implemented these algorithms in a prototype system, and evaluated their performance using extensive experiments in the context of multimedia stream analysis. The results show that our Greedy algorithm consistently outperforms other known algorithms under various settings and scales well as the numbers of queries and filters increase.

Predictable performance and high query concurrency for data analytics

“…Finally, work sharing has been investigated extensively in the context of streaming database systems [3,8,9,[19][20][21]. By sharing work (or state) among continuous query operators, a streaming DBMS can maintain a low per-tuple processing cost and thus handle a large number of continuous queries over fast streams.…”

An Adaptive Ordering Framework for Filtering Multimedia Streams

“…The most important metric is the evaluation time, which is defined as the average time taken by our prototype system to evaluate one image item against all filtering rules in current system. We implement the greedy algorithm named ECBG which is proposed by [3] and EPABS for comparison.…”

“…Then [3] improved upon those of [2], [3] proposed an edge-coverage based greedy algorithm whose performance was better than [2] under various settings. Our problem also generalizes the shared ordering problem [1,2,3] as well as the well-studied pipelined filter ordering problem [4,5,6,7]. Pipelined filtering ordering problem has been considered in [4,5,6,1].…”

“…In such a problem, many filtering rules are posed on the streams to specify the filtering objectives, and many filtering units, which are low-level semantic detectors trained offline using tagged examples, are used to evaluate the filtering rules. As shown in Figure 1, there are three filtering rules (i.e., R 1 ,R 2 and R 3 ) and four filtering units (i.e., U 1 ,U 2 ,U 3 and U 4 ). The objective is to evaluate all the filtering rules in a low time cost by properly ordering all the filtering units.…”

“…In paper [2], they introduced the shared filtering unit ordering problem and present query coverage based greedy algorithm. In paper [3], they took this problem as covering the edges between filtering rules and filtering units through a suitable choice of filtering units. However, despite these greedy algorithms consider all these factors, they fix the selectivity by prior knowledge, and this strategy can't reflex local information of multimedia stream.…”