Using structural contexts to compress semistructured text collections

Abstract

We describe a compression model for semistructured documents, called Structural Contexts Model (SCM), which takes advantage of the context information usually implicit in the structure of the text. The idea is to use a separate model to compress the text that lies inside each different structure type (e.g., different XML tag). The intuition behind SCM is that the distribution of all the texts that belong to a given structure type should be similar, and different from that of other structure types.We mainly focus on semistatic models, and test our idea using a word-based Huffman method. This is the standard for compressing large natural language text databases, because random access, partial decompression, and direct search of the compressed collection is possible. This variant, dubbed SCMHuff, retains those features and improves Huffman's compression ratios.We consider the possibility that storing separate models may not pay off if the distribution of different structure types is not different enough, and present a heuristic to merge models with the aim of minimizing the total size of the compressed database. This gives an additional improvement over the plain technique. The comparison against existing prototypes shows that, among the methods that permit random access to the collection, SCMHuff achieves the best compression ratios, 2-4% better than the closest alternative.From a purely compression-aimed perspective, we combine SCM with PPM modeling. A separate PPM model is used to compress the text that lies inside each different structure type. The result, SCMPPM, does not permit random access nor direct search in the compressed text, but it gives 2-5% better compression ratios than other techniques for texts longer than 5 megabytes.

Schema-Based Compression of XML Data with Relax NG

“…Compression of XML has been studied from a variety of perspectives. Some researchers aim to achieve minimal size [3], [4], [5], others focus on efficient streaming [6], [7], [8] -a balance between bandwidth and encode/decode times -and still others answer XML queries directly from compressed representations [9]. Representations that support queries are necessarily larger than those that do not; Ferragina et al [10] report increases of 25 to 96% compared to opaque representations.…”

“…They provide benefits similar to those of XMill's containers. Adiego et al [5] augment this technique with a heuristic to combine certain context models; this yields better results on some large data sets. In our experiments, the performance of xmlppm was nearly always competitive.…”

“…gz refers to the GZIPOutputStream that is part of the Java API; lzma is the Lempel-Ziv Markov chain Algorithm by Igor Pavlov; 3 bz2 is the bzip2 algorithm by Julian Seward (Java implementation by Keiron Liddle); 4 ppm is Prediction by Partial Match [19] implemented by Bob Carpenter. 5 -Z none|gz|lzma|bz2|ppm -specifies what kind of compression to apply to the stream containing the data (text) elements. For many data sets, this is the most significant setting.…”