Fast on-line index construction by geometric partitioning

Lester, Nicholas; Moffat, Alistair; Zobel, Justin

doi:10.1145/1099554.1099739

Cited by 42 publications

(49 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Logarithmic Merge for base k = 2). The results (reported in Table 1) are consistent with earlier findings [6,1] and show that the off-line method really should not be used in a dynamic environment. Logarithmic merge exhibits indexing performance close to off-line index construction and query processing performance close to Immediate Merge.…”

Section: Resultssupporting

confidence: 90%

“…This quadratic time complexity renders Immediate Merge infeasible for text collections much larger than the available main memory. Recently, Büttcher and Clarke [1] and Lester et al [6] have proposed mergebased update strategies that do not share this shortcoming. By allowing a controlled number of on-disk indices to exist in parallel, indexing efficiency is greatly increased, while query processing performance remains almost unchanged compared to the Immediate Merge strategy.…”

Section: Merge-based Index Maintenancementioning

confidence: 99%

See 1 more Smart Citation

A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

Büttcher

Clarke

2006

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In-place and merge-based index maintenance are the two main competing strategies for on-line index construction in dynamic information retrieval systems based on inverted lists. Motivated by recent results for both strategies, we investigate possible combinations of in-place and merge-based index maintenance. We present a hybrid approach in which long posting lists are updated in-place, while short lists are updated using a merge strategy. Our experimental results show that this hybrid approach achieves better indexing performance than either method (in-place, merge-based) alone.

show abstract

Section: Resultssupporting

confidence: 90%

Section: Merge-based Index Maintenancementioning

confidence: 99%

A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

Büttcher

Clarke

2006

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Moreover, according to the observation in Figure 3, the attribute frequency distribution follows the Zipf law [37] like style. This interesting observation motivates us to explore the successful inverted index [7,22,25,33,35,38,39] used in information retrieval to manipulate the dataspaces.…”

Section: Sparse Featuresmentioning

confidence: 99%

“…Lester [22] partitioning the index for efficient on-line index. To make documents immediately accessible, the index is divided into a controlled number of partitions.…”

Section: Related Workmentioning

confidence: 99%

Indexing dataspaces with partitions

Song

Chen

2012

World Wide Web

View full text Add to dashboard Cite

Dataspaces are recently proposed to manage heterogeneous data, with features like partially unstructured, high dimension and extremely sparse. The inverted index has been previously extended to retrieve dataspaces. In order to achieve more efficient access to dataspaces, in this paper, we first introduce our survey of data features in the real dataspaces. Based on the features observed in our study, several partitioning based index approaches are proposed to accelerate the query processing in dataspaces. Specifically, the vertical partitioning index utilizes the partitions on tokens to merge and compress data. We can both reduce the number of I/O reads and avoid aggregation of data inside a compressed list. The horizontal partitioning index supports pruning partitions of tuples in the top-k query. Thus, we can reduce the computation overhead of irrelevant candidate tuples to the query. Finally, we also propose a hybrid index with both vertical and horizontal partitioning. The extensive experiment results in real data sets demonstrate that our approaches outperform the previous techniques and scale well with the large data size.

show abstract

“…If query requests were permitted using the old index, recall and precision of the search engine would not be ensured since the web set has changed. Nowadays, most search engines use incremental index updating strategy [2]. This paper is related to the Internet connection among clusters.…”

Section: Introductionmentioning

confidence: 99%

A Two-Tier Distributed Full-Text Indexing System

Zhang¹,

Chen²,

He³

et al. 2014

Appl. Math. Inf. Sci.

View full text Add to dashboard Cite

Abstract:The performance of indexing systems is very important for a search engine. Usually, indexing systems on large-scale clusters can provide high search efficiency, but it brings expensive hardware costs. The costs would be greatly reduced if a distributed indexing system runs on small-scale clusters connected by the Internet. Two current inverted file partitioning schemes: document partitioning and term partitioning, have their merits individually. A two-tier distributed full-text indexing system is implemented, which uses document partitioning among the clusters and term partitioning inside each cluster. Our experiments show that the system performs well in search efficiency, resource consuming and load balance.

show abstract

Fast on-line index construction by geometric partitioning

Cited by 42 publications

References 16 publications

A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

A Hybrid Approach to Index Maintenance in Dynamic Text Retrieval Systems

Indexing dataspaces with partitions

A Two-Tier Distributed Full-Text Indexing System

Contact Info

Product

Resources

About