Boosting Text Compression with Word-Based Statistical Encoding

Abstract

Semistatic word-based byte-oriented compressors are known to be attractive alternatives to compress natural language texts. With compression ratios around 30-35%, they allow fast direct searching of compressed text. In this article we reveal that these compressors have even more benefits. We show that most of the state-of-the-art compressors benefit from compressing not the original text, but the compressed representation obtained by a word-based byte-oriented statistical compressor. For example, p7zip with a dense-coding preprocessing achieves even better compression ratios and much faster compression than p7zip alone. We reach compression ratios below 17% in typical large English texts, which was obtained only by the slow PPM compressors. Furthermore, searches perform much faster if the final compressor operates over word-based compressed text. We show that typical self-indexes also profit from our preprocessing step. They achieve much better space and time performance when indexing is preceded by a compression step. Apart from using the well-known Tagged Huffman code, we present a new suffix-free Dense-Code-based compressor that compresses slightly better. We also show how some self-indexes can handle non-suffix-free codes. As a result, the compressed/indexed text requires around 35% of the space of the original text and allows indexed searches for both words and phrases. * A preliminary partial version of this work appeared in [1].

On the Randomness of Compressed Data

“…. entries, up to a predetermined maximal size, say 2 18 . There are several options to continue, like restarting from scratch with 9 bits, or considering the dictionary as static and not adjoining any more strings.…”

“…It is therefore natural to assume that once as much redundancy as possible has been removed, the remaining text should be indistinguishable from random data, for if some more regularities can be detected, they could be targeted in an additional round of compression. Indeed, Fariña et al [18] focus on byte-oriented word based compressors for natural languages, and show that such compressed files can be further compressed using any general purpose compressor such as gzip. They note that the frequencies of the byte values generated by a byte-oriented, word based, compressor, are far from uniform, as opposed to the output of arithmetic coding.…”

FM-index for Dummies

“…Before applying the BWT, we encode the text using a dense code. First we use the (s, c, b, o)-DC (SCBDC) [10] scheme, which is both prefix-and suffix-free and thus requires no verifications. We use this code on nybbles, i.e., set the parameters in a way to have s + c + b + o = 16.…”

A Novel Text Processing for Better Compression and Security in Cloud

“…The authors in [5] exploit word based byte-oriented compression, and then transit text through character positioned compression. They use end-tagged dense code, as it is easier to build than a Huffman code.…”