Memory efficient ranking

Moffat, Alistair; Zobel, Justin; Sacks-Davis, Ron

doi:10.1016/0306-4573(94)90002-7

Cited by 25 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A better method is to use low-precision approximations to the document weights, which can reduce each document length to around six bits without significantly affecting retrieval effectiveness or retrieval time [27]. Furthermore, in a multi-user environment the cost of storing the weights can be amortised over all active processes, since the weights are static and can be stored in shared memory.…”

Section: Ranked Query Evaluationmentioning

confidence: 99%

Self-indexing inverted files for fast text retrieval

Moffat

Zobel

1996

ACM Trans. Inf. Syst.

Self Cite

294

236

View full text Add to dashboard Cite

Query processing costs on large text databases are dominated by the need to retrieve and scan the inverted list of each query term. Here we show that query response time for conjunctive Boolean queries and for informal ranked queries can be dramatically reduced, at little cost in terms of storage, by the inclusion of an internal index in each inverted list. This method has been applied in a retrieval system for a collection of nearly two million short documents. Our experimental results show that the selfindexing strategy adds less than 20% to the size of the inverted file, but, for Boolean queries of 5-10 terms, can reduce processing time to under one fifth of the previous cost. Similarly, ranked queries of 40-50 terms can be evaluated in as little as 25% of the previous time, with little or no loss of retrieval effectiveness.

show abstract

Section: Ranked Query Evaluationmentioning

confidence: 99%

Self-indexing inverted files for fast text retrieval

Moffat

Zobel

1996

ACM Trans. Inf. Syst.

Self Cite

294

236

View full text Add to dashboard Cite

show abstract

“…Problematically the impact values are not integers. Consequently, impact values are quantized into integers [8,2]; in practice a simple linear scaling to fit in one byte is effective.…”

Section: Related Workmentioning

confidence: 99%

“…Postings lists can be very long and can require a substantial amount of processing time. Various techniques have been developed to manage their size including compression [10,4] and static pruning; and various techniques have been developed to decrease processing time including impact ordering [8] and dynamic pruning [2]. However, little attention has been give to the very large number of short postings lists typically seen in an inverted index.…”

Section: Introductionmentioning

confidence: 99%

Managing short postings lists

Trotman

Jia

Crane

2013

Proceedings of the 18th Australasian Document Computing Symposium

View full text Add to dashboard Cite

Previous work has examined space saving and throughput increasing techniques for long postings lists in an inverted file search engine. In this contribution we show that highly sporadic terms (terms that occur in 1 or 2 documents) are a high proportion of the unique terms in the collection and that these terms are seen in queries. The previously known space saving method of storing their short postings lists in the vocabulary is compared to storing in the postings file. We quantify the saving as about 6.5%, with no loss in precision, and suggest the adoption of this technique.

show abstract

“…Moffat et al [4] observed that approximations to components of the ranking function were as effective as exact values (the document length can be approximate). Anh et al [1] observed that the term / document weight could be precomputed and stored in the impact ordered index; during indexing they evaluated the ranking function for every term with respect to every document and stored that result rather than term frequency.…”

Section: Introductionmentioning

confidence: 99%

“…However, such pre-calculated values are floating point and do not compress well, so Moffat et al [4] quantized these scores into b-bit integers. Several approximations were explored by Anh et al [1], who show that either a linear mapping, or alternatively skewing to provide extra granularity at the lower scores works well.…”

Section: Introductionmentioning

confidence: 99%

Maintaining discriminatory power in quantized indexes

Crane

Trotman

O’Keefe

2013

Proceedings of the 22nd ACM International Conference on Conference on Information &Amp; Knowledge Management - CIKM '13

View full text Add to dashboard Cite

The time cost of searching with an inverted index is directly proportional to the number of postings processed and the cost of processing each posting. Dynamic pruning reduces the number of postings examined. Pre-calculation then quantization of term / document weights reduces the cost of evaluating each posting. The effect of quantization on precision, latency, and index size is examined herein. We show empirically that there is an ideal size (in bits) for storing the quantized scores. Increasing this adversely affects index size and search latency; decreasing it adversely affects precision. We observe a relationship between the collection size and ideal quantization size, and provide a way to determine the number of bits to use from the collection size.

show abstract

Memory efficient ranking

Cited by 25 publications

References 17 publications

Self-indexing inverted files for fast text retrieval

Self-indexing inverted files for fast text retrieval

Managing short postings lists

Maintaining discriminatory power in quantized indexes

Contact Info

Product

Resources

About