Optimized processing of multiple aggregate continuous queries

Abstract

Data Streams Management Systems are designed to support monitoring applications which require the processing of hundreds of Aggregate Continuous Queries (ACQs). These ACQs typically have different time granularities, with possibly different selection predicates and group-by attributes. In order to achieve scalability in the presence of heavy workloads, in this paper, we introduce the concept of 'Weaveability' as an indicator of the potential gains of sharing the processing of ACQs. We then propose Weave Share, a cost-based optimizer that exploits weaveability to optimize the shared processing of ACQs. Our experimental analysis shows that Weave Share outperforms the alternative sharing schemes generating up to four orders of magnitude better quality plans. Finally, we describe a practical implementation of the Weave Share optimizer.

Three-Level Processing of Multiple Aggregate Continuous Queries

“…Similarly, our cost-based Weave Share multiple ACQs optimizer [12], realized in the AQSIOS prototype [6], [3], adopts TwoOps. Like Shared Time Slices, Weave Share addresses the problem of shared processing of ACQs with varying windows.…”

“…Orthogonally, the Weave Share [12] optimizer has been proposed to utilize the Paired Window technique in order to minimize the cost of the final aggregation. Weave Share selectively groups the ACQs into multiple execution trees.…”

“…Under Weave Share, each tree employs its own sub-aggregation operator, which is shared between the ACQs in that tree (as shown in Figure 2). In order to decide the best grouping of ACQs, Weave Share introduces the concept of weaveability [12], which measures the potential gains from the shared processing of ACQs and weaves, i.e., groups together, the ACQs based on their weaveability measures.…”

“…Similarly to all previous works in optimizing the execution of ACQs [16], [31], [21], [12], we assume that each of these ACQs is not part of a larger complex CQ and is only characterized by pre-aggregation predicates, group-by attributes and window settings. The predicates can be arbitrary complex.…”

“…TriOps is a new aggregate operator implementation that works in synergy with the new Weave Share optimizer [12] to minimize the total cost of processing multiple ACQs. TriOps employs a three-level data processing model that minimizes the repetition of operations at the sub-aggregation level.…”