Space-Efficient Frameworks for Top- k String Retrieval

Abstract

The inverted index is the backbone of modern web search engines. For each word in a collection of web documents, the index records the list of documents where this word occurs. Given a set of query words, the job of a search engine is to output a ranked list of the most relevant documents containing the query. However, if the query consists of an arbitrary string-which can be a partial word, multiword phrase, or more generally any sequence of characters-then word boundaries are no longer relevant and we need a different approach. In string retrieval settings, we are given a set D = {d 1 , d 2 , d 3 , . . . , d D } of D strings with n characters in total taken from an alphabet set = [σ ], and the task of the search engine, for a given query pattern P of length p, is to report the "most relevant" strings in D containing P. The query may also consist of two or more patterns. The notion of relevance can be captured by a function score(P, d r ), which indicates how relevant document d r is to the pattern P. Some example score functions are the frequency of pattern occurrences, proximity between pattern occurrences, or pattern-independent PageRank of the document.The first formal framework to study such kinds of retrieval problems was given by Muthukrishnan [SODA 2002]. He considered two metrics for relevance: frequency and proximity. He took a threshold-based approach on these metrics and gave data structures that use O(n log n) words of space. We study this problem in a somewhat more natural top-k framework. Here, k is a part of the query, and the top k most relevant (highestscoring) documents are to be reported in sorted order of score. We present the first linear-space framework (i.e., using O(n) words of space) that is capable of handling arbitrary score functions with near-optimal O( p + k log k) query time. The query time can be made optimal O( p + k) if sorted order is not necessary. Further, we derive compact space and succinct space indexes (for some specific score functions). This space compression comes at the cost of higher query time. At last, we extend our framework to handle the case of multiple patterns. Apart from providing a robust framework, our results also improve many earlier results in index space or query time or both.

Time-Optimal Top-$k$ Document Retrieval

“…It is also worth mentioning that the journal version of the original paper of Hon et al has recently appeared as well [40]. Here they show how to obtain O(p + k) time if the top-k results are not to be returned sorted by relevance.…”

Efficient Representation of Multidimensional Data over Hierarchical Domains

“…For example, categorical range counting queries (i.e., count the number of different values in a range) requires in general Ω(log n/ log log n) time if using O(n polylog n) space [11], where n is the array size, but if queries form a hierarchy it is easily solved in constant time and O(n) bits [13]. A second example is the range mode problem (i.e., find the most frequent value in a range), which is believed to require time Ω(n 1.188 ) if using O(n 1.188 ) space [4], but if queries form a hierarchy it is easily solved in constant time and linear space [8].…”

“…In this paper we aim at a compact data structure to represent data cubes where the domains in each dimension are hierarchical. Following the general idea of the tailored solutions to the problems we mentioned [13,8], our structure partitions the space according to the hierarchies, instead of performing a regular partition like generic multidimensional structures. Therefore, the queries of interest for OLAP applications, which combine nodes of the different hierarchies, will require aggregating the information of just a few nodes in our partitions, much fewer than if we used a generic space partitioning method.…”