Optimal Join Algorithms Meet Top-k

Abstract

Top-k queries have been studied intensively in the database community and they are an important means to reduce query cost when only the "best" or "most interesting" results are needed instead of the full output. While some optimality results exist, e.g., the famous Threshold Algorithm, they hold only in a fairly limited model of computation that does not account for the cost incurred by large intermediate results and hence is not aligned with typical database-optimizer cost models. On the other hand, the idea of avoiding large intermediate results is arguably the main goal of recent work on optimal join algorithms, which uses the standard RAM model of computation to determine algorithm complexity. This research has created a lot of excitement due to its promise of reducing the time complexity of join queries with cycles, but it has mostly focused on full-output computation. We argue that the two areas can and should be studied from a unified point of view in order to achieve optimality in the common model of computation for a very general class of top-k-style join queries. This tutorial has two main objectives. First, we will explore and contrast the main assumptions, concepts, and algorithmic achievements of the two research areas. Second, we will cover recent, as well as some older, approaches that emerged at the intersection to support efficient ranked enumeration of join-query results. These are related to classic work on k-shortest path algorithms and more general optimization problems, some of which dates back to the 1950s. We demonstrate that this line of research warrants renewed attention in the challenging context of ranked enumeration for general join queries.

Beyond Equi-joins: Ranking, Enumeration and Factorization

“…Since this algorithm assumes restricted key-to-key joins, later works generalize it to more general joins [37,46,58,84], which may even involve theta-joins [60]. However, they retain the middleware cost model, hence do not provide any non-trivial guarantees when the actual join cost is taken into account [79]. Ilyas et al [47] survey some of these approaches, along with some related ones such as building top-𝑘 indexes [22,77] or views [25,43].…”

“…Ranked enumeration. Requiring join answers to be returned in a specific order gives rise to ranked enumeration [29,31,78,79,85,86], where the main goal is to quickly return the most important answers without having to materialize and sort the entire output. Ranked enumeration generalizes the well-known top-𝑘 paradigm, removing the requirement of having to specify 𝑘 in advance.…”

“…Ranked enumeration generalizes the well-known top-𝑘 paradigm, removing the requirement of having to specify 𝑘 in advance. Besides, non-trivial complexity guarantees for top-𝑘 join algorithms [47] had been derived only for the "middleware" cost model of the celebrated Threshold Algorithm [36], which only accounts for the number of distinct data items accessed [79]. In contrast, work on enumeration, including this paper, accounts for all the steps taken by the algorithm, using the standard RAM model of computation.…”

Any-k Algorithms for Enumerating Ranked Answers to Conjunctive Queries

“…A ranked enumeration-or "any-𝑘"-algorithm needs to perform well for all possible values of 𝑘. Any-𝑘 algorithms can conceptually be seen as a fusion of top-𝑘 [77] and anytime algorithms [131] that gradually improve their result over time. Secondly and more importantly, algorithms for top-𝑘 joins, including the famous Threshold Algorithm [58], were developed for a middleware-centric cost model that only accounts for accesses to external data sources, and not for the cost of potentially huge intermediate results [121]. In contrast, our goals are strict guarantees for the time and memory for every 𝑘 th ranked answer in the standard RAM-model of computation.…”

Efficient Constant-Factor Approximate Enumeration of Minimal Subsets for Monotone Properties with Weight Constraints

“…: spanning trees [26], s-t paths [28,35], s-t cuts [47], and (weighted) perfect matchings [38] (See also [20], for more information). Among others, enumerating multiple solutions with some specific order has attracted special interests in the field of database research, which they call ranked enumeration [18,41,46]. When we wish to enumerate K solutions in the non-decreasing order of their cardinality, this is equivalent to K-best enumeration.…”