Compressing relations and indexes

Goldstein, Jonathan; Ramakrishnan, R.; Shaft, Uri

doi:10.1109/icde.1998.655800

Cited by 114 publications

(97 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, the list-adaptive codecs demonstrated a good tradeoff in inverted index size and query response time. As our best practice recommendation, we found that the simpler Frame of Reference (FOR) [24] codec results in fast average response times for all types of posting list payload, slightly outperforming the more complex but state-of-the-art PFOR implementations. Finally, we note that the gains obtainable from the compression of the various information within a posting list are cumulative.…”

Section: Discussionmentioning

confidence: 99%

“…Indeed, the contributions of this paper are three-fold: Firstly, we separately analyse the compression benefits of document ids, term frequencies, field frequencies and positions, in terms of average query response time and index size; Secondly, we analyse how different term distributions caused by anchor text affect the achievable compression; Finally, we use these thorough experiments on two standard corpora, namely GOV2 and ClueWeb09 to derive best practices for index compression. The results of our study demonstrate that the contribution of compression in time-and space-efficiency varies greatly depending on the type of posting payload information, while leading us to suggest the use of the simpler Frame of reference (FOR) [24] list-adaptive algorithms for the best benefit to average query response times. This paper is structured as follows: Section 2 provides a background on efficient ranked retrieval; Section 3 summarises the compression codecs we analyse later within an IR system; Section 4 describes our experimental setup.…”

Section: Introductionmentioning

confidence: 97%

“…For example, in a word we can store {509, 510, 511} as three 9-bits values, with the highest 4 bits of the word reflecting this configuration, at a cost of one wasted bit. Frame of reference (FOR): Proposed by Goldstein et al [24], FOR compresses integers in blocks of fixed size (e.g. 128 elements).…”

Section: List-adaptive Codecsmentioning

confidence: 99%

See 2 more Smart Citations

On Inverted Index Compression for Search Engine Efficiency

Catena

Macdonald

Ounis

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Efficient access to the inverted index data structure is a key aspect for a search engine to achieve fast response times to users' queries. While the performance of an information retrieval (IR) system can be enhanced through the compression of its posting lists, there is little recent work in the literature that thoroughly compares and analyses the performance of modern integer compression schemes across different types of posting information (document ids, frequencies, positions). In this paper, we experiment with different modern integer compression algorithms, integrating these into a modern IR system. Through comprehensive experiments conducted on two large, widely used document corpora and large query sets, our results show the benefit of compression for different types of posting information to the space-and time-efficiency of the search engine. Overall, we find that the simple Frame of Reference compression scheme results in the best query response times for all types of posting information. Moreover, we observe that the frequency and position posting information in Web corpora that have large volumes of anchor text are more challenging to compress, yet compression is beneficial in reducing average query response times.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

On Inverted Index Compression for Search Engine Efficiency

Catena

Macdonald

Ounis

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In the past years, compression techniques have focussed on CPU optimised compression algorithms [64,65,66,67]. It has been shown in [65,66,67] that the decompression performance depends on the complexity of the execution flow of the algorithm.…”

Section: High-performance Compression For Node Indexing Schemementioning

confidence: 99%

“…FOR determines the range of possible values in a block, called a frame, and maps each value into this range by storing just enough bits to distinguish between the values [64]. Given a frame [min, max], FOR needs log 2 (max − min + 1) bits, that we call bit frame in the rest of the paper, to encode each integer in a block.…”

Section: Frame Of Reference (For)mentioning

confidence: 99%

Searching web data: An entity retrieval and high-performance indexing model

Delbru

Campinas

Tummarello

2012

Journal of Web Semantics

View full text Add to dashboard Cite

Utilizing the column imprints to accelerate no‐partitioning hash joins in large‐scale edge systems

Liu

2020

Trans Emerging Tel Tech

View full text Add to dashboard Cite

With the increasing number of edge devices in large-scale edge systems, more and more data are collected to be processed. In such big data scenarios, there is a resurgence of interest in main-memory analytic databases because of the large RAM capacity of modern servers and the increasing demand for real-time analytic platforms. In such databases, join is at the heart of almost every query plan. Join also stays as a time-consuming operation when the denormalization overhead is too large to be applicable. However, the current implementations of these operations have not fully leveraged the new features (eg, SIMD, multi-core) provided by the modern hardware. The goal of this article is to design efficient algorithms for joins by judiciously exploiting every bit of RAM and all the available parallelisms in each processing unit. For join operations, hash joins have been studied, improved, and reexamined over decades.In this article, we propose to utilize a secondary index to improve hash joins without the physical partitioning. Specifically, in the build phase, the hash values are scattered evenly into the logical partitions of the hash table; in the probe phase, the secondary index is used as the hints to re-order the probing sequence, such that the locality of the hash probing is increased. We benchmark the performance of the proposed techniques in our column-store research prototype. Extensive experiments on the synthetic data and the real data show that our methods offer significant performance improvement over their counterparts.

show abstract

Compressing relations and indexes

Cited by 114 publications

References 9 publications

On Inverted Index Compression for Search Engine Efficiency

On Inverted Index Compression for Search Engine Efficiency

Searching web data: An entity retrieval and high-performance indexing model

Utilizing the column imprints to accelerate no‐partitioning hash joins in large‐scale edge systems

Contact Info

Product

Resources

About