Low-cost management of inverted files for online full-text search

Abstract

In dynamic environments with frequent content updates, we require online full-text search that scales to large data collections and achieves low search latency. Several recent methods that support fast incremental indexing of documents typically keep on disk multiple partial index structures that they continuously update as new documents are added. However, spreading indexing information across multiple locations on disk tends to considerably decrease the search responsiveness of the system. In the present paper, we take a fresh look at the problem of online full-text search with consideration of the architectural features of modern systems. Selective Range Flush is a greedy method that we introduce to manage the index in the system by using fixed-size blocks to organize the data on disk and dynamically keep low the cost of data transfer between memory and disk. As we experimentally demonstrate with the Proteus prototype implementation that we developed, we retrieve indexing information at latency that matches the lowest achieved by existing methods. Additionally, we reduce the total building cost by 30% in comparison to methods with similar retrieval time.

Search in social networks with access control

“…The accumulated postings are routinely combined with the rest of the data in a hierarchy based on geometric partitioning [8]. More advanced techniques exist [7,9], but they would not change the tradeoffs that we show. We use PForDelta [15] for compression, which has shown efficient decompression performance in recent studies [13].…”

Mimir: a term-distributed retrieval system for secret documents

“…The variety of cipher terms caused by key change will result in index update. For index update, Margaritis and Anastasiadis (2009) only flush selectively the terms with most posting lists in memory into disk to merge it with primary index when the memory gets full with new posting lists. Gurajada and Kumar (2009) propose a new merge-based index maintenance strategy for information retrieval systems.…”

Dynamic memory allocation policies for postings in real-time Twitter search

“…In the buffer-and-flush approach, Margaritis and Anastasiadis [12] present an interesting alternative beyond the three strategies discussed above. They make a slightly different design choice: when the in-memory buffer reaches capacity, instead of flushing the entire in-memory index, they choose to flush only a portion of the term space (a contiguous range of terms based on lexicographic sort order), performing a merge with the corresponding on-disk portions of the inverted lists.…”