Space-Efficient Algorithms for Document Retrieval

Abstract

Abstract. We study the Document Listing problem, where a collection D of documents d1, . . . , d k of total length i di = n is to be preprocessed, so that one can later efficiently list all the ndoc documents containing a given query pattern P of length m as a substring. Muthukrishnan (SODA 2002) gave an optimal solution to the problem; with O(n) time preprocessing, one can answer the queries in O(m + ndoc) time. In this paper, we improve the space-requirement of the Muthukrishnan's solution from O(n log n) bits to |CSA| + 2n + n log k(1 + o(1)) bits, where |CSA| ≤ n log |Σ|(1 + o (1)) is the size of any suitable compressed suffix array (CSA), and Σ is the underlying alphabet of documents. The time requirement depends on the CSA used, but we can obtain e.g. the optimal O(m+ndoc) time when |Σ|, k = O(polylog(n)). For general |Σ|, k the time requirement becomes O(m log |Σ| + ndoc log k). Sadakane (ISAAC 2002) has developed a similar space-efficient variant of the Muthukrishnan's solution; we obtain a better time requirement in most cases, but a slightly worse space requirement.

Top-k Ranked Document Search in General Text Databases

“…We include three baseline methods derived from previous work on the document listing problem. The first two are implementations of Välimäki and Mäkinen [22] and Sadakane [20] as described in Section 3, labelled VM and Sada respectively. The third, ℓ-gram, is a close variant of Puglisi et al's inverted index of ℓ-grams [16], used with parameters ℓ = 3 and block size= 4096.…”

“…By representing D with a wavelet tree, values C[i] can be calculated on demand, rather than stored explicitly [22]. This reduces the space to | CSA | + n log N + 2n + o(n log N ) bits, where | CSA | is the size of any compressed suffix array and N is the number of documents (Section 2).…”

Colored Range Queries and Document Retrieval

“…He used |CSA| + 4n + o(n) additional bits for data structures to compute the pattern's frequency in each document, increasing the time bound to O(search(m) + ndoc(lookup(n) + log log ndoc)) (assuming lookup(n) is also the time to find CSA −1 [ ], where CSA −1 is the inverse permutation). Välimäki and Mäkinen [37] gave an alternative slower-but-smaller version of Muthukrishnan's CRL data structure, in which they used a 2n + o(n) bit, O(1) time RMQ succinct index due to Fischer and Heun [13] that requires access to C. Välimäki and Mäkinen showed how access to C can be implemented by rank and select queries on S; specifically, for 1 ≤ ≤ n,…”

“…The space bound is the sum of the space bounds and the time bound per reported color is O(t acc + t enum + t rank ), the latter term for computing frequencies. For example, 2+9: is Välimäki and Mäkinen's scheme [37]. 1: is the scheme by Gagie, Puglisi, and Turpin [15].…”

“…In this paper, motivated by problems in document retrieval, we consider the latter three kinds of problems, which are often referred to as "colored" range queries: colored range listing (with or without color frequencies), colored range top-k queries, and colored range counting. These have been associated, respectively, to very relevant document retrieval queries on general texts [31,35,37,20,15,12,9]: listing the documents where a pattern appears (possibly computing term frequencies), finding the most relevant documents to a query (under a tf × idf scheme, for example), and computing document frequencies. Such techniques have been shown to be competitive [9], even beating classical inverted indexes on natural-language texts.…”