Intra-query Concurrent Pipelined Processing for Distributed Full-Text Retrieval

Jonassen, Simon; Bratsberg, Svein Erik

doi:10.1007/978-3-642-28997-2_35

Cited by 9 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The optimization splits the document ID range into several sub-ranges, called fragments, and does intra-query parallelization at the fragment level, both across the nodes and on the same node. The experiments [6] with a smaller subset of the TREC 2005 Efficiency Topics indicated that this optimization allows to reach a similar peak-throughput at nearly half of the latency. We assume this to be applicable to our current method, however a further evaluation for the 2006 topics is needed.…”

Section: Conclusion and Further Workmentioning

confidence: 96%

“…One of these, intra-query concurrent pipelined processing, has already been evaluated in our recent work [6]. The preliminary baseline of [6] is an early version of the Max-Score optimization we are about to present, MSD s , and the results of [6] are therefore directly applicable to this method.…”

Section: Related Workmentioning

confidence: 99%

“…Additionally, it opens a possibility for dynamic load balancing with low repartitioning overhead and hybrid query processing. Finally, different from [4,5,6], our experiments use two query logs with very different characteristics, and a varied collection size.…”

Section: Related Workmentioning

confidence: 99%

“…This feature can be utilized to provide intra-query concurrency and improve the performance at low multiprogramming levels. In [6], we have already evaluated an extension to an earlier version of MSD S . The optimization splits the document ID range into several sub-ranges, called fragments, and does intra-query parallelization at the fragment level, both across the nodes and on the same node.…”

Section: Conclusion and Further Workmentioning

confidence: 99%

See 3 more Smart Citations

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

Jonassen

Bratsberg

2013

World Wide Web

Self Cite

View full text Add to dashboard Cite

Abstract. Web search engines need to provide high throughput and short query latency. Recent results show that pipelined query processing over a term-wise partitioned inverted index may have superior throughput. However, the query processing latency and scalability with respect to the collections size are the main challenges associated with this method. In this paper, we evaluate the effect of inverted index skipping on the performance of pipelined query processing. Further, we introduce a novel idea of using Max-Score pruning within pipelined query processing and a new term assignment heuristic, partitioning by Max-Score. Our current results indicate a significant improvement over the state-of-the-art approach and lead to several further optimizations, which include dynamic load balancing, intra-query concurrent processing and a hybrid combination between pipelined and non-pipelined execution.

show abstract

Section: Conclusion and Further Workmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Conclusion and Further Workmentioning

confidence: 99%

See 2 more Smart Citations

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

Jonassen

Bratsberg

2013

World Wide Web

Self Cite

View full text Add to dashboard Cite

show abstract

“…The pipelined query processing scheme is originally proposed by Moffat et al [2007] and is later enhanced by Jonassen and Bratsberg [2010, 2012a, 2012b. This scheme solves the bottleneck problem at the central broker and can also improve the computational load balance of the system.…”

Section: Pipelined Query Processing Schemementioning

confidence: 98%

A term-based inverted index partitioning model for efficient distributed query processing

et al. 2013

Self Cite

View full text Add to dashboard Cite

In a shared-nothing, distributed text retrieval system, queries are processed over an inverted index that is partitioned among a number of index servers. In practice, the index is either document-based or term-based partitioned. This choice is made depending on the properties of the underlying hardware infrastructure, query traffic distribution, and some performance and availability constraints. In query processing on retrieval systems that adopt a term-based index partitioning strategy, the high communication overhead due to the transfer of large amounts of data from the index servers forms a major performance bottleneck, deteriorating the scalability of the entire distributed retrieval system. In this work, to alleviate this problem, we propose a novel inverted index partitioning model that relies on hypergraph partitioning. In the proposed model, concurrently accessed index entries are assigned to the same index servers, based on the inverted index access patterns extracted from the past query logs. The model aims tominimize the communication overhead that will be incurred by future queries while maintaining the computational load balance among the index servers. We evaluate the performance of the proposed model through extensive experiments using a real-life text collection and a search query sample. Our results show that considerable performance gains can be achieved relative to the term-based index partitioning strategies previously proposed in literature. In most cases, however, the performance remains inferior to that attained by document-based partitioning. © 2013 ACM

show abstract

Decoding billions of integers per second through vectorization

Lemire

Boytsov

2013

Softw. Pract. Exper.

216

183

View full text Add to dashboard Cite

SUMMARYIn many important applications-such as search engines and relational database systems-data are stored in the form of arrays of integers. Encoding and, most importantly, decoding of these arrays consumes considerable CPU time. Therefore, substantial effort has been made to reduce costs associated with compression and decompression. In particular, researchers have exploited the superscalar nature of modern processors and single-instruction, multiple-data (SIMD) instructions. Nevertheless, we introduce a novel vectorized scheme called SIMD-BP128? that improves over previously proposed vectorized approaches. It is nearly twice as fast as the previously fastest schemes on desktop processors (varint-G8IU and PFOR). At the same time, SIMD-BP128? saves up to 2 bits/int. For even better compression, we propose another new vectorized scheme (SIMD-FastPFOR) that has a compression ratio within 10% of a state-of-the-art scheme (Simple8b) while being two times faster during decoding.

show abstract

Intra-query Concurrent Pipelined Processing for Distributed Full-Text Retrieval

Cited by 9 publications

References 17 publications

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

A term-based inverted index partitioning model for efficient distributed query processing

Decoding billions of integers per second through vectorization

Contact Info

Product

Resources

About