Algebraic query optimization for distributed top-k queries

Abstract

Distributed top-k query processing is increasingly becoming an essential functionality in a large number of emerging application classes. This paper addresses the efficient algebraic optimization of top-k queries in wide-area distributed data repositories where the index lists for the attribute values (or text terms) of a query are distributed across a number of data peers and the computational costs include network latency, bandwidth consumption, and local peer work. We use a dynamic programming approach to find the optimal execution plan using compact data synopses for selectivity estimation that is the basis for our cost model. The optimized query is executed in a hierarchical way involving a small and fixed number of communication phases. We have performed experiments on real web data that show the benefits of distributed top-k query optimization both in network resource consumption and query response time

Smooth Interpolating Histograms with Error Guarantees

“…We use the optimization framework described in [6], as the experiments in the paper operate on the same data set, using the official GOV benchmark queries. For the comparison, we optimize the queries using different cardinality estimators, execute them, and then compare the resulting runtime and network traffic.…”

“…Yu et al [5] present a modification of TPUT where the range bounds are adapted to the specifics of the value distributions. Along these lines stands our own work [6] on distributed top-k query optimization. The basic idea behind the algorithms presented in [4,5,7] is the transformation of a top-k query into the union of range queries where the range is determined by an initial retrieval phase.…”