Filtered document retrieval with frequency‐sorted indexes

Persin, Michael; Zobel, Justin; Sacks-Davis, Ron

doi:10.1002/(sici)1097-4571(199610)47:10<749::aid-asi3>3.3.co;2-u

Cited by 76 publications

(91 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If approximate results are acceptable, the TAAT approach can be made even more efficient using impact ordering [1,2,33]. The key idea of impact ordering is to precompute the TF for each document a term appears in.…”

Section: Term-at-a-time Processing (Taat)mentioning

confidence: 99%

Efficient in-memory top-k document retrieval

Culpepper

Petri

Scholer

2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

For over forty years the dominant data structure for ranked document retrieval has been the inverted index. Inverted indexes are effective for a variety of document retrieval tasks, and particularly efficient for large data collection scenarios that require disk access and storage. However, many efficiency-bound search tasks can now easily be supported entirely in-memory as a result of recent hardware advances.In this paper we present a hybrid algorithmic framework for inmemory bag-of-words ranked document retrieval using a self-index derived from the FM-Index, wavelet tree, and the compressed suffix tree data structures, and evaluate the various algorithmic trade-offs for performing efficient queries entirely in-memory. We compare our approach with two classic approaches to bag-of-words queries using inverted indexes, term-at-a-time (TAAT) and document-at-atime (DAAT) query processing. We show that our framework is competitive with state-of-the-art indexing structures, and describe new capabilities provided by our algorithms that can be leveraged by future systems to improve effectiveness and efficiency for a variety of fundamental search operations.

show abstract

Section: Term-at-a-time Processing (Taat)mentioning

confidence: 99%

Efficient in-memory top-k document retrieval

Culpepper

Petri

Scholer

2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…There has been a significant amount of work in the IR and database communities on this issue under various scenarios; see [1,2,12,14,16,24,28,35] for recent work. Various schemes are apparently in use in current engines but details are closely guarded.…”

Section: Term-based Rankingmentioning

confidence: 99%

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Long¹,

Suel²

2006

World Wide Web

View full text Add to dashboard Cite

Large web search engines have to answer thousands of queries per second with interactive response times. Due to the sizes of the data sets involved, often in the range of multiple terabytes, a single query may require the processing of hundreds of megabytes or more of index data. To keep up with this immense workload, large search engines employ clusters of hundreds or thousands of machines, and a number of techniques such as caching, index compression, and index and query pruning are used to improve scalability. In particular, two-level caching techniques cache results of repeated identical queries at the frontend, while index data for frequently used query terms are cached in each node at a lower level.We propose and evaluate a three-level caching scheme that adds an intermediate level of caching for additional performance gains. This intermediate level attempts to exploit frequently occurring pairs of terms by caching intersections or projections of the corresponding inverted lists. We propose and study several offline and online algorithms for the resulting weighted caching problem, which turns out to be surprisingly rich in structure. Our experimental evaluation based on a large web crawl and real search engine query log shows significant performance gains for the best schemes, both in isolation and in combination with the other caching levels. We also observe that a careful selection of cache admission and eviction policies is crucial for best overall performance.

show abstract

“…The use of filters [22] is an evaluation technique that allows to recognize what documents are probable of being ranked, reducing the main memory volume required and the queries evaluation time, without harming the effectiveness of results. Because of that, we use the filtering technique proposed by Persin [21] to discard the irrelevant documents for the queries.…”

Section: Distributed Inverted Files and Vector Space Modelmentioning

confidence: 99%