Efficient online index maintenance for contiguous inverted lists

Lester, Nicholas; Zobel, Justin; Williams, Hugh E.

doi:10.1016/j.ipm.2005.09.005

Cited by 51 publications

(36 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The need for contiguity makes it necessary to relocate the lists when they run out of empty space at their end [17,18]. One can amortize the cost of relocation by preallocating list space for future appends using various criteria [25].…”

Section: Related Workmentioning

confidence: 99%

“…The latest related methods amortize the cost by permitting the creation of multiple inverted files on disk and merging them according to specific patterns [15,16]. Even though in-place index maintenance has linear asymptotic disk cost that is lower than the polynomial cost of merge-based methods, merge-based methods are experimentally shown to use sequential disk transfers and outperform inplace methods [17].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Low-cost management of inverted files for online full-text search

Margaritis

Anastasiadis

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

In dynamic environments with frequent content updates, we require online full-text search that scales to large data collections and achieves low search latency. Several recent methods that support fast incremental indexing of documents typically keep on disk multiple partial index structures that they continuously update as new documents are added. However, spreading indexing information across multiple locations on disk tends to considerably decrease the search responsiveness of the system. In the present paper, we take a fresh look at the problem of online full-text search with consideration of the architectural features of modern systems. Selective Range Flush is a greedy method that we introduce to manage the index in the system by using fixed-size blocks to organize the data on disk and dynamically keep low the cost of data transfer between memory and disk. As we experimentally demonstrate with the Proteus prototype implementation that we developed, we retrieve indexing information at latency that matches the lowest achieved by existing methods. Additionally, we reduce the total building cost by 30% in comparison to methods with similar retrieval time.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Low-cost management of inverted files for online full-text search

Margaritis

Anastasiadis

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…To tackle this problem, Shoens et al [13] (and also Shieh et al [12]) proposed a predictive over-allocation strategy for improving the performance of In-place indexing model. And Lester et al [10] proposed another solution by maintaining an inverted file for each term. However, the problem of fragmentation is left to the file system.…”

Section: In-place Updatementioning

confidence: 99%

On-line index maintenance using horizontal partitioning

Gurajada

2009

Proceedings of the 18th ACM Conference on Information and Knowledge Management

View full text Add to dashboard Cite

In this paper, we propose a new merge-based index maintenance strategy for Information Retrieval systems. The new model is based on partitioning of the inverted index across the terms in it. We exploit the query log to partition the on-disk inverted index into two types of sub-indexes. Inverted lists of the terms contained in the queries that are frequently posed to the Information Retrieval systems are kept in one partition, called frequent-term index and the other inverted lists form another partition, called infrequentterm index. We use a lazy-merge strategy for maintaining infrequent-term sub-indexes, and an active merge strategy for maintaining frequent-term sub-indexes. The sub-indexes are also similarly split into frequent and in-frequent parts. Experimental results show that the proposed method improves both index maintenance performance and query performance compared to the existing merge-based strategies.

show abstract

“…More recently, Luk and Lam [23] describe an internal-memory inverted file implementation based on hash tables with chaining, but their method also does not support fast item removals. Lester et al [21,22] and Büttcher et al [10] describe externalmemory inverted file implementations that support item insertions only. Büttcher and Clarke [9] consider trade-offs for allowing for both item insertions and removals, and Guo et al [17] give a solution for performing such operations by using a B-tree variant.…”

Section: Introductionmentioning

confidence: 99%

Cache-Oblivious Dictionaries and Multimaps with Negligible Failure Probability

Goodrich

Hirschberg

Mitzenmacher

et al. 2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. A dictionary (or map) is a key-value store that requires all keys be unique, and a multimap is a key-value store that allows for multiple values to be associated with the same key. We design hashing-based indexing schemes for dictionaries and multimaps that achieve worst-case optimal performance for lookups and updates, with minimal space overhead and sub-polynomial probability that the data structure will require a rehash operation. Our dictionary structure is designed for the Random Access Machine (RAM) model, while our multimap implementation is designed for the cache-oblivious external memory (I/O) model. The failure probabilities for our structures are sub-polynomial, which can be useful in cryptographic or data-intensive applications.

show abstract

Efficient online index maintenance for contiguous inverted lists

Cited by 51 publications

References 22 publications

Low-cost management of inverted files for online full-text search

Low-cost management of inverted files for online full-text search

On-line index maintenance using horizontal partitioning

Cache-Oblivious Dictionaries and Multimaps with Negligible Failure Probability

Contact Info

Product

Resources

About