New space/time tradeoffs for top- k document retrieval on sequences

Abstract

We address the problem of indexing a collection D = {T 1 , T 2 , ...T D } of D string documents of total length n, so that we can efficiently answer top-k queries: retrieve k documents most relevant to a pattern P of length p given at query time.There exist linear-space data structures, that is, using O(n) words, that answer such queries in optimal O(p + k) time for an ample set of notions of relevance. However, using linear space is not sufficiently good for large text collections. In this paper we explore how far the space/time tradeoff for this problem can be pushed. We obtain three results: (1) When relevance is measured as term frequency (number of times P appears in a document T i ), an index occupying |CSA|+o(n) bits answers the query in time O(t search (p)+k lg 2 k lg ε n), where CSA is a compressed suffix array indexing D, t search is its time to find the suffix array interval of P, and ε > 0 is any constant. (2) With the same measure of relevance, an index occupying |CSA| + n lg D + o(n lg σ + n lg D) bits answers the query in time O(t search (p) + k lg * k), where lg * k is the iterated logarithm of k. (3) When the relevance depends only on the documents, an index occupying |CSA| + O(n lg lg n) bits answers the query in O(t search (p) + k t SA ) time, where t SA is the time the CSA needs to retrieve a suffix array cell. On our way, we obtain some other results of independent interest.

Time-Optimal Top-$k$ Document Retrieval

“…Another trend adds to the space the so-called document array [51], which uses n lg D + o(n log D) bits and enables faster solutions. Currently the fastest one achieves time O(p + k log * k) [56]. This is very close to optimal, but not yet our O(p/ log σ n + k) time.…”

“…We have shown that our structure can use, instead, O(n(log σ + log D)) bits for the tf measure (and slightly more for others), but the constants are still large. There is a whole trend of reduced-space representations for general document retrieval problems with the tf measure [64,70,41,21,32,38,12,31,68,39,56]. The current situation is as follows [52]: One trend aims at the least space usage.…”

“…The current situation is as follows [52]: One trend aims at the least space usage. It has managed to use just D lg(n/D) + O(D) + o(n) bits on top of a compressed suffix array of the collection, and the best time complexity it has achieved is O(p + k log 2 k log 1+ε n) for any constant ε > 0 [56]. Another trend adds to the space the so-called document array [51], which uses n lg D + o(n log D) bits and enables faster solutions.…”

Gapped Indexing for Consecutive Occurrences

“…A well studied line of work within document indexing is document indexing for top-k queries [12,23,24,25,26,33,34,37,39,42,43]. The goal is to efficiently report the top-k documents of smallest weight, where the weight is a function of the query.…”

Top-k Term-Proximity in Succinct Space

“…Hon et al achieved O(t s (p)+k t SA log 3+ε n) query time, using O(n/ log ε n) bits. Subsequent work (see [19,25]) improved the initial result up to O(t s (p) + k t SA log 2 k log ε n) [23], and also considered compact indexes, which may use o(n log n) bits on top of the CSA. For example, these achieve O(t s (p) + k t SA log k log ε n) query time using n log σ + o(n) further bits [12], or O(t s (p) + k log * k) query time using n log D + o(n log n) further bits [24].…”