Efficient exploitation of similar subexpressions for query processing

Zhou, Jingren; Larson, Per-Åke; Freytag, Johann-Christoph; Lehner, Wolfgang

doi:10.1145/1247480.1247540

Cited by 82 publications

(61 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selecting queries to materialize and reuse during processing has spanned fine-grained approaches from subqueries [27] and partial materialization [20,28], to coarse-grained methods as part of multiquery optimization and common subexpressions [16,35]. Picking views from a workload of queries typically uses the AND-OR graph representations from multiquery optimization [16,27], or adopts signature and subsumption methods for common subexpressions [35].…”

Section: Materialization and Query Optimization Strategiesmentioning

confidence: 99%

“…Picking views from a workload of queries typically uses the AND-OR graph representations from multiquery optimization [16,27], or adopts signature and subsumption methods for common subexpressions [35]. [27] picks additional views to materialize amongst subqueries of the view definition, but only performs first-order maintenance and does not consider the full framework (binding patterns, etc) required with higher-order deltas.…”

Section: Materialization and Query Optimization Strategiesmentioning

confidence: 99%

See 1 more Smart Citation

DBToaster: higher-order delta processing for dynamic, frequently fresh views

et al. 2014

View full text Add to dashboard Cite

Applications ranging from algorithmic trading to scientific data analysis require realtime analytics based on views over databases that change at very high rates. Such views have to be kept fresh at low maintenance cost and latencies. At the same time, these views have to support classical SQL, rather than window semantics, to enable applications that combine current with aged or historical data.In this paper, we present viewlet transforms, a recursive finite differencing technique applied to queries. The viewlet transform materializes a query and a set of its higher-order deltas as views. These views support each other's incremental maintenance, leading to a reduced overall view maintenance cost. The viewlet transform of a query admits efficient evaluation, the elimination of certain expensive query operations, and aggressive parallelization. We develop viewlet transforms into a workable query execution technique, present a heuristic and cost-based optimization framework, and report on experiments with a prototype dynamic data management system that combines viewlet transforms with an optimizing compilation technique. The system supports tens of thousands of complete view refreshes a second for a wide range of queries.

show abstract

Section: Materialization and Query Optimization Strategiesmentioning

confidence: 99%

Section: Materialization and Query Optimization Strategiesmentioning

confidence: 99%

DBToaster: higher-order delta processing for dynamic, frequently fresh views

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Studies include efficient detection of sharing opportunities across queries [12], and search algorithms for finding efficient query execution plans that materialize the common intermediate results for reuse [11]. Our shared optimization extensions build on similar techniques while the goal is to improve communication efficiency.…”

Section: Related Workmentioning

confidence: 99%

Plan-based complex event detection across distributed sources

2008

View full text Add to dashboard Cite

Complex Event Detection (CED) is emerging as a key capability for many monitoring applications such as intrusion detection, sensorbased activity & phenomena tracking, and network monitoring. Existing CED solutions commonly assume centralized availability and processing of all relevant events, and thus incur significant overhead in distributed settings. In this paper, we present and evaluate communication efficient techniques that can efficiently perform CED across distributed event sources.Our techniques are plan-based: we generate multi-step event acquisition and processing plans that leverage temporal relationships among events and event occurrence statistics to minimize event transmission costs, while meeting application-specific latency expectations. We present an optimal but exponential-time dynamic programming algorithm and two polynomial-time heuristic algorithms, as well as their extensions for detecting multiple complex events with common sub-expressions. We characterize the behavior and performance of our solutions via extensive experimentation on synthetic and real-world data sets using our prototype implementation.

show abstract

“…Recent work by Zhou et al [28] introduces a novel technique for detecting common subexpressions which allows MQO to be integrated in production systems. The experiments show that applying MQO across a few tens of queries is extremely efficient.…”

Section: Multi Query Optimizationmentioning

confidence: 99%

Shared workload optimization

et al. 2014

View full text Add to dashboard Cite

As a result of increases in both the query load and the data managed, as well as changes in hardware architecture (multicore), the last years have seen a shift from query-at-a-time approaches towards shared work (SW) systems where queries are executed in groups. Such groups share operators like scans and joins, leading to systems that process hundreds to thousands of queries in one go.SW systems range from storage engines that use in-memory cooperative scans to more complex query processing engines that share joins over analytical and star schema queries. In all cases, they rely on either single query optimizers, predicate sharing, or on manually generated plans. In this paper we explore the problem of shared workload optimization (SWO) for SW systems. The challenge in doing so is that the optimization has to be done for the entire workload and that results in a class of stochastic knapsack with uncertain weights optimization, which can only be addressed with heuristics to achieve a reasonable runtime. In this paper we focus on hash joins and shared scans and present a first algorithm capable of optimizing the execution of entire workloads by deriving a global executing plan for all the queries in the system. We evaluate the optimizer over the TPC-W and the TPC-H benchmarks. The results prove the feasibility of this approach and demonstrate the performance gains that can be obtained from SW systems.

show abstract

Efficient exploitation of similar subexpressions for query processing

Cited by 82 publications

References 19 publications

DBToaster: higher-order delta processing for dynamic, frequently fresh views

DBToaster: higher-order delta processing for dynamic, frequently fresh views

Plan-based complex event detection across distributed sources

Shared workload optimization

Contact Info

Product

Resources

About