Time-Optimal Top-$k$ Document Retrieval

Abstract

Let D be a collection of D documents, which are strings over an alphabet of size σ, of total length n. We describe a data structure that uses linear space and and reports k most relevant documents that contain a query pattern P , which is a string of length p packed in p/ log σ n words, in time O(p/ log σ n + k). This is optimal in the RAM model in the general case where log D = Θ(log n), and involves a novel RAM-optimal suffix tree search. Our construction supports an ample set of important relevance measures, such as the number of times P appears in a document (called term frequency), a fixed document importance, and the minimal distance between two occurrences of P in a document. When log D = o(log n), we show how to reduce the space of the data structure from O(n log n) to O(n(log σ + log D + log log n)) bits, and to O(n(log σ + log D)) bits in the case of the popular term frequency measure of relevance, at the price of an additive term O(log ε σ n) in the query time, for any constant ε > 0. We also consider the dynamic scenario, where documents can be inserted and deleted from the collection. We obtain linear space and query time O(p(log log n) 2 / log σ n+log n+k log log k), whereas insertions and deletions require O(log 1+ε n) time per symbol, for any constant ε > 0. Finally, we consider an extended static scenario where an extra parameter par(P, d) is defined, and the query must retrieve only documents d such that par(P, d) ∈ [τ 1 , τ 2 ], where this range is specified at query time. We solve these queries using linear space and O(p/ log σ n + log 1+ε n + k log ε n) time, for any constant ε > 0. Our technique is to translate these top-k problems into multidimensional geometric search problems. As a bonus, we describe some improvements to those problems.

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

“…By using perfect hashing to store the first characters of the edge labels descending from each node of v, we reach the locus in optimal time O(m) and the space is still O(n). If P comes packed using w/ log σ symbols per computer word, we can descend in time O( m log(σ)/w ) [91], which is optimal in the packed model. In the suffix array, all the suffixes starting with P form a range SA[sp..ep], which can be binary searched in time O(m log n), or O(m + log n) with additional structures [81].…”

“…We then replace our trie by a more sophisticated structure, which is described by Navarro and Nekrich [91,Sec. 2], built on the O(rs) distinct strings of length s. Let d = w/ log σ .…”

Fast Compressed Self-indexes with Deterministic Linear-Time Construction

“…In the RAM model with word size Θ(log n), and if the consecutive symbols of P come packed into |P |/ log σ n words, the optimal time is instead O(|P |/ log σ n). This optimal time was recently reached by Navarro and Nekrich [31] (note that their time is not optimal if w = ω(log n)), with a simple application of weak-prefix search, already hinted in the original article [2]. However, even the randomized construction time of the weak-prefix search structure is O(n log n), for any constant > 0.…”

“…Compared with previous work, other indexes may be faster at counting, but either they are not built in linear deterministic time [5,19,31] or they are not compressed [31,7]. Our index outperforms all the previous compressed [13,1,6], as well as some uncompressed [15], indexes that can be built deterministically.…”

“…As an application of our tools, we also show that an index using O(n log σ) bits of space can be built in linear deterministic time, so that it can count in time O(|P |/ log σ n + log n(log log n) 2 ), which is RAM-optimal for w = Θ(log n) and sufficiently long patterns. Current indexes obtaining similar counting time require O(n log σ) construction time [19] or higher [31], or O(n log n) bits of space [31,7].…”