Bounds on the entropy of patterns of I.I.D. sequences

Abstract

Abstract-Bounds on the entropy of patterns of sequences generated by independently identically distributed (i.i.d.) sources are derived. A pattern is a sequence of indices that contains all consecutive integer indices in increasing order of first occurrence. If the alphabet of a source that generated a sequence is unknown, the inevitable cost of coding the unknown alphabet symbols can be exploited to create the pattern of the sequence. This pattern can in turn be compressed by itself. The bounds derived here are functions of the i.i.d. source entropy, alphabet size, and letter probabilities. It is shown that for large alphabets, the pattern entropy must decrease from the i.i.d. one. The decrease is in many cases more significant than the universal coding redundancy bounds derived in prior works. The pattern entropy is confined between two bounds that depend on the arrangement of the letter probabilities in the probability space. For very large alphabets whose size may be greater than the coded pattern length, all low probability letters are packed into one symbol. The pattern entropy is upper and lower bounded in terms of the i.i.d. entropy of the new packed alphabet. Correction terms, which are usually negligible, are provided for both upper and lower bounds.

Universal Lossless Compression With Unknown Alphabets—The Average Case

“…Furthermore, better universal compression performance is also possible in the case where the alphabet size k is sublinear in n or even fixed. Moreover, even better non-universal compression is sometimes possible because every pattern represents a collection of many sequences, thus reducing the overall pattern entropy (see, e.g., [31], [34], [36], [38], [39]). …”

Pattern entropy - revisited

“…Since # % ¥ S R ¡ is the result of data processing, its entropy must be no greater than . Subsequently to the results in [14], it was independently shown in [4] and [8] that for discrete i.i.d. sources, the pattern entropy rate is equal to that of the underlying i.i.d.…”

“…The first results on pattern entropy in [11], [14], [16], however, showed that for sufficiently large alphabets, the pattern block entropy must decrease from the i.i.d. one even more significantly than the universal coding penalty for coding patterns.…”

A Lower-Bound for the Maximin Redundancy in Pattern Coding

“…It was first introduced byÅberg in [8] as a solution to the multi-alphabet coding problem, where the message x contains only a small subset of the known alphabet A. It was further studied and motivated in a series of articles by Shamir [9][10][11][12] and by Jevtić, Orlitsky, Santhanam and Zhang [13][14][15][16] for practical applications: the alphabet is unknown and has to be transmitted separately anyway (for instance, transmission of a text in an unknown language), or the alphabet is very large in comparison to the message (consider the case of images with k = 2 24 colors, or texts when taking words as the alphabet units).…”

“…To explain the notion of pattern, let us take the example of [9]: string x = "abracadabra" is made of n = 11 characters. The information it conveys can be separated in two blocks:…”