Multi-route query processing and optimization

Nehme, Rimma V.; Works, Karen; Lei, Chuan; Rundensteiner, Elke A.; Bertino, Elisa

doi:10.1016/j.jcss.2012.09.010

Cited by 21 publications

(25 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By ignoring strong inter-stream correlations existing optimizers miss the critical join pair pruning opportunity illustrated above. To avoid enumerating the entire search space, they identify partitions either by employing off-the-shelf randomized partitioning algorithms [13] or by utilizing greedy search algorithms without backtracking [20]. These simplistic search heuristics tend to result in ineffective optimization decisions as confirmed by our experiments (Sec.…”

Section: Introductionmentioning

confidence: 52%

“…Like Query Mesh [13], CMR is also a multi-route approach. Once an effective multi-route solution has been identified by CMR, it can be successfully executed by any multi-route infrastructure, including the Query Mesh executor.…”

Section: Related Workmentioning

confidence: 99%

“…However, real-world data is often skewed with possibly complex correlations hidden in both a single stream or across multiple streams [16]. Thus using overall stream statistics as the base for constructing one single query plan serving all data tuples in the stream misses important opportunities for effective query optimization [14,13].…”

Section: Introductionmentioning

confidence: 99%

“…Finding an effective multi-route solution which fully utilizes the optimization opportunities illustrated above is challenging. First as shown in [13] given a single stream S the complexity of dividing S tuples into partitions with distinct optimal execution plans is already exponential in the number of S's domain values even when only considering S's local data distribution. Worst yet to utilize the join pair pruning, the partitioning of one stream must also consider the partition decisions made for the other participating streams.…”

Section: Introductionmentioning

confidence: 99%

“…While initial work on multi-route processing has recently emerged [13,20], the existing techniques do not address the above described challenges. To reduce the search space these multi-route optimizers construct partitions by either only considering the local statistics of each individual stream [13] in isolation or by going with the unrealistic simplification of assuming conditional independence across attributes [20]. By ignoring strong inter-stream correlations existing optimizers miss the critical join pair pruning opportunity illustrated above.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

High performance stream query processing with correlation-aware partitioning

Cao

Rundensteiner

2013

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

State-of-the-art optimizers produce one single optimal query plan for all stream data, in spite of such a singleton plan typically being sub-optimal or even poor for highly correlated data. Recently a new stream processing paradigm, called multi-route approach, has emerged as a promising approach for tackling this problem. Multiroute first divides data streams into several partitions and then creates a separate query plan for each combination of partitions. Unfortunately current approaches suffer from severe shortcomings, in particular, the lack of an effective partitioning strategy and the prohibitive query optimization expense. In this work we propose the first practical multi-route optimizer named correlation-aware multi-route stream query optimizer (or CMR) that solves both problems. By exploiting both intra-and inter-stream correlations of streams, CMR produces effective partitions without having to undertake repeated expensive query plan generation. The produced partitions not only are best served by distinct optimal query plans, but also leverage the partition-driven pruning opportunity. Experimental results with both synthetic and real life stream data confirm that CMR outperforms the state-of-the-art solutions up to an order of magnitude in both the query optimization time and the run-time execution performance.

show abstract

Section: Introductionmentioning

confidence: 52%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

High performance stream query processing with correlation-aware partitioning

Cao

Rundensteiner

2013

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

show abstract

Cloud‐based data streams optimization

Najib

Ismail

Badr

et al. 2018

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Many modern applications of sensor networks and transaction analysis require real‐time processing of their stream data sets. These data streams vary continuously over time. Current stream processing approaches focus on only one of the two optimization perspectives, proposing optimization techniques for data streams processing regardless of the processing environment or improving the processing environment only. In this paper, a brief survey of recent approaches to data streams processing coming from the two optimizations perspectives is proposed; their shortcomings are presented as well. Then, a proposal to an innovative and integrative framework is developed; it is referred to as the continuous query optimization based on multiple plans (CQOMP) for data streams over the cloud environment. CQOMP combines the two optimization perspectives and provides an optimized stream clusters processing using multiple split query plans. Each plan is constructed for a cluster of data that has nearest characteristics and it processes streams tuples over the cloud. We also propose a novel algorithm called the optimized multiple plans (OMP) for processing data streams clusters on Cloud Computing. The OMP algorithm efficiently divides data streams and generates optimized multiple split plans. Each plan is for processing a group of data streams on the cloud. We present the experimental results of the OMP solution compared to the alternative state‐of‐the‐art data stream approaches. The experiments show the efficiency and the scalability of the combined OMP algorithm on different cloud environments, the real Amazon cloud environment, and the simulated windows azure cloud environment. This article is categorized under: Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Classification Technologies > Data Preprocessing Technologies > Structure Discovery and Clustering

show abstract