Improved Single-Term Top-<i>k</i> Document Retrieval

Gog, Simon; Navarro, Gonzalo

doi:10.1137/1.9781611973754.3

Cited by 6 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We first explore the new DOC REP representation separately and then plug it into the single-term version of the SUccinct Retrieval Framework (SURF) [6] to evaluate query performance in a search engine.…”

Section: Methodsmentioning

confidence: 99%

“…• The bit vector H requires less than 2n − N + o(n) bits (see [6]) and is used to map the lexicographic range into the x-range in G in constant time.…”

Section: The Basic Framework and Data Structuresmentioning

confidence: 99%

“…In this section we will first discuss the mapping from the lexicographic range to the grid from [6] in more detail and also present the necessary basic data structures. We then explain how these data structures can be used to replace the explicit (or direct) representation of the repetition array.…”

Section: An Indirect Representation Of the Repetition Arraymentioning

confidence: 99%

“…Konow and Navarro [11] and Gog and Navarro [6] created a practical version of the linear space index which only requires 2.5 − 4.0 times the space of the original collection and can still answer top-10 queries well below one millisecond. The engineering of Navarro and Nekrich's initial idea did not only result in practical indexes but also further simplified their original proposal.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Elias-Fano meets Single-Term Top-k Document Retrieval

Labeit¹,

Gog²

2017

2017 Proceedings of the Ninteenth Workshop on Algorithm Engineering and Experiments (ALENEX)

Self Cite

View full text Add to dashboard Cite

A fundamental problem in Information Retrieval is to determine the k most relevant documents of a collection for a given query word or phrase P . In a recent result, Navarro and Nekrich [SODA 2012] showed that this problem can be solved in optimal time complexity of O(|P | + k) with a precomputed linear-space index. The size of this optimal-time index was estimated to be 80 times the collection size, rendering it not to be practical. In subsequent work, Navarro and Konow [DCC 2013] and Gog and Navarro [ALENEX 2015] created a practical version with slightly worse query time guarantees but reduced the space to 2.5 − 3 times the collection size. The index is conceptually simple and is divided in five components. In this paper we show how the n log N bits required by the usually largest component -the so called repetition array -can be reduced to n log log n + O(n), where n is the size of the collection and N the number of documents. As the overall query time complexity matches the one of the old index, we achieve a theoretically superior time-space trade-off. We explore the practical properties of the improved index in a detailed experimental study and compare to the previously established baseline. Index sizes are now between 1.5 − 2 times the collection size while query speed is comparable to the larger indexes. We also show that the new approach automatically adapts to highly repetitive text collections, which are for instance produced by version control systems.

show abstract

Section: Methodsmentioning

confidence: 99%

“…• The bit vector H requires less than 2n − N + o(n) bits (see [6]) and is used to map the lexicographic range into the x-range in G in constant time.…”

Section: The Basic Framework and Data Structuresmentioning

confidence: 99%

Section: An Indirect Representation Of the Repetition Arraymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Elias-Fano meets Single-Term Top-k Document Retrieval

Labeit¹,

Gog²

2017

2017 Proceedings of the Ninteenth Workshop on Algorithm Engineering and Experiments (ALENEX)

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the case of static succinct data structures, trees [4], web and social graphs [9], and document collections to support efficient queries [13] have been implemented and tested. However, the current landscape of implemented dynamic succinct data structures is quite limited.…”

Section: Related Workmentioning

confidence: 99%

A General Framework for Dynamic Succinct and Compressed Data Structures

Klitzke¹,

Nicholson

2015

2016 Proceedings of the Eighteenth Workshop on Algorithm Engineering and Experiments (ALENEX)

View full text Add to dashboard Cite

Succinct data structures are becoming increasingly popular in big data processing applications due to their low memory consumption. However, a feature that is currently lacking from most implementations of succinct data structures is dynamism. In this paper we design, implement, and test a general framework that allows for practical dynamic succinct structures. Firstly, a key component of our approach is careful memory management, which is often overlooked in the succinct data structures literature. Most succinct data structures allocate and deallocate relatively small data blocks each time a modify, insert, or delete operation occurs. We demonstrate experimentally that the space cost of neglecting memory management can be over 25% for dynamic data structures of this type. Secondly, using our memory management approach, we describe implementations of compressed modifiable bit vectors, and extended compressed random access memory (recently proposed by Jansson, Sadakane, and Sung [ICALP 2012]). Finally, we implement and test our data structures using several popular compression libraries, and both synthetic data (for the compressed modifiable bit vector) and a real-world temporal graph (for the extended compressed random access memory). Our data structures provide an easy to use interface that allow standard algorithms (in our example, breadth-first search in a graph) to be

show abstract

Compact Indexes for Flexible Top-$$k$$ Retrieval

Gog

Petri

2015

Combinatorial Pattern Matching

Self Cite

View full text Add to dashboard Cite

We engineer a self-index based retrieval system capable of rank-safe evaluation of top-k queries. The framework generalizes the GREEDY approach of Culpepper et al. (ESA 2010) to handle multiterm queries, including over phrases. We propose two techniques which significantly reduce the ranking time for a wide range of popular Information Retrieval (IR) relevance measures, such as TF×IDF and BM25. First, we reorder elements in the document array according to document weight. Second, we introduce the repetition array, which generalizes Sadakane's (JDA 2007) document frequency structure to document subsets. Combining document and repetition array, we achieve attractive functionality-space trade-offs. We provide an extensive evaluation of our system on terabyte-sized IR collections.

show abstract

Improved Single-Term Top-k Document Retrieval

Cited by 6 publications

References 24 publications

Elias-Fano meets Single-Term Top-k Document Retrieval

Elias-Fano meets Single-Term Top-k Document Retrieval

A General Framework for Dynamic Succinct and Compressed Data Structures

Compact Indexes for Flexible Top-$$k$$ Retrieval

Contact Info

Product

Resources

About