Any-k Algorithms for Enumerating Ranked Answers to Conjunctive Queries

Abstract

We study ranked enumeration for Conjunctive Queries (CQs) where the answers are ordered by a given ranking function (e.g., an ORDER BY clause in SQL). We develop "any-k" algorithms which, without knowing the number 𝑘 of desired answers, push the ranking into joins and avoid materializing the join output earlier than necessary. For this to be possible, the ranking function needs to obey a certain kind of monotonicity; the supported ranking functions include the common sum-of-weights case where query answers are compared by sums of input weights, as well as any commutative selective dioid. One core insight of our work is that the problem is closely related to the fundamental task of path enumeration in a weighted DAG. We generalize and improve upon classic research on finding the 𝑘 th shortest path and unify into the same framework several solutions from different areas that had been studied in isolation. For the time to the 𝑘 th ranked CQ answer (for every value of 𝑘), our approach is optimal in data complexity precisely for the same class of queries where unranked enumeration is optimal-and only slower by a logarithmic factor. In a more careful analysis of combined complexity, we uncover a previously unknown tradeoff between two different any-k algorithms: one has lower complexity when the number of returned results is small, the other when the number is very large. This tradeoff is eliminated under a stricter monotonicity property that we exploit to design a novel algorithm that asymptotically dominates all previously known alternatives, including the well-known algorithm of Eppstein for sum-of-weights path enumeration. We empirically demonstrate the findings of our theoretical analysis in an experimental study that highlights the superiority of our approach over the join-then-rank approach that existing database systems typically follow.

Efficient Computation of Quantiles over Joins

“…Let ⊎ be multiset union. An (aggregate) ranking function is subset-monotone [23] if agg 𝑤 (𝐿 1 ) ⪯ agg 𝑤 (𝐿 2 ) implies that agg 𝑤 (𝐿 ⊎ 𝐿 1 ) ⪯ agg 𝑤 (𝐿 ⊎ 𝐿 2 ) for all multisets 𝐿, 𝐿 1 , 𝐿 2 . All ranking functions we consider in this work have this property.…”

“…All ranking functions we consider in this work have this property. We note that subset-monotonicity has been used as an assumption in ranked enumeration [15,23] and is a stronger requirement than the more well-known monotonicity notion of Fagin et al [10].…”

“…We begin by showing that we can select a pivot in linear time for every acyclic join query and every ranking function that satisfies a monotonicity assumption (also used in the problem of ranked enumeration [15,23]), which all functions considered here satisfy. Note that the assumption of acyclicity is required since, otherwise, it is impossible to even determine whether the join query has any answer in quasilinear time, under conventional conjectures in finegrained complexity [6].…”