On Inverted Index Compression for Search Engine Efficiency

Abstract

Abstract. Efficient access to the inverted index data structure is a key aspect for a search engine to achieve fast response times to users' queries. While the performance of an information retrieval (IR) system can be enhanced through the compression of its posting lists, there is little recent work in the literature that thoroughly compares and analyses the performance of modern integer compression schemes across different types of posting information (document ids, frequencies, positions). In this paper, we experiment with different modern integer compression algorithms, integrating these into a modern IR system. Through comprehensive experiments conducted on two large, widely used document corpora and large query sets, our results show the benefit of compression for different types of posting information to the space-and time-efficiency of the search engine. Overall, we find that the simple Frame of Reference compression scheme results in the best query response times for all types of posting information. Moreover, we observe that the frequency and position posting information in Web corpora that have large volumes of anchor text are more challenging to compress, yet compression is beneficial in reducing average query response times.

SIMD compression and the intersection of sorted integers

“…A format inspired by FASTPFOR (where bits from exceptions are bit packed) is a part of the search engine Apache Lucene as of version 4.5. In contrast, Catena et al found that FASTPFOR provided no response time benefit compared with NewPFD and OptPFD when compressing document identifiers. Possibly, the discrepancy can be explained by the fact that they divided their arrays into small chunks of 1024 integers prior to compression.…”

Compression, SIMD, and Postings Lists

“…The experiments of Catena et al [3] 6 , discussed in Section 1, use the Hadoop implementation, which can also store negative numbers. Positive integers less than 128 are stored in one byte with a leading (high bit) 0.…”

“…Recently Catena et al [3] re-examined the result of Scholer et al in light of hardware, operating system, compiler, language, and search engine algorithmic improvements over the last 10 years. They show that, of the schemes they tested, the PForDelta family of schemes resulted in the highest throughput.…”

Peningkatan Kinerja Pencarian Dokumen Tugas Akhir Menggunakan Porter Stemmer Bahasa Indonesia dan Fungsi Peringkat Okapi BM25

“…Struktur data pengindeksan information retrieval yang terkenal adalah inverted index. Untuk setiap kata yang muncul di koleksi dokumen, inverted index akan berisi posting list atau daftar dokumen yang mempunyai kata tersebut yang berisi informasi kemunculan kata tersebut pada dokumen (dapat berupa frekuensi kata atau posisi) [7].…”