Efficient Compressed Inverted Index Skipping for Disjunctive Text-Queries

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

“…In a recent work [5], we have investigated these techniques for a non-distributed index. In particular, we have presented a self-skipping inverted index designed specifically for NewPFoR [14].…”

“…Additionally, it opens a possibility for dynamic load balancing with low repartitioning overhead and hybrid query processing. Finally, different from [4,5,6], our experiments use two query logs with very different characteristics, and a varied collection size.…”

“…We use Algorithm 1 to describe the following processing step performed by the query processor. The space-limited pruning method itself has been originally presented by Lester et al [7] and the skipping optimization for a non-distributed index has already been presented in our previous work [5]. Therefore, the goal of this section is to describe the improvements to pipelined query processing.…”

“…Therefore, the goal of this section is to describe the improvements to pipelined query processing. However, for the sake of intelligibility we also explain the most important details derived from the prior work [5,7].…”

“…We apply the layout described in our previous work [5]. With the basic index, each posting list is divided into groups of 128 entries, which are stored as two chunks containing 128 document ID deltas and 128 frequencies, both compressed with NewPFoR.…”