A Lower-Bound for the Maximin Redundancy in Pattern Coding

Garivier, Aurélien

doi:10.3390/e11040634

Cited by 9 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This remarkably simple method is however expected to have a poor performance when α is large. Indeed, it is proved in Garivier (2006) that R + Ψ (Ψ 1:n ) is lower-bounded by 1.84 n log n 1 3 (see also Shamir (2006) and references therein), which indicates that pattern coding is probably suboptimal as soon as α is larger than 3.…”

Section: A Pattern Codingmentioning

confidence: 99%

Coding on Countably Infinite Alphabets

Boucheron

Garivier

Gassiat

2009

IEEE Trans. Inform. Theory

Self Cite

111

View full text Add to dashboard Cite

This paper describes universal lossless coding strategies for compressing sources on countably infinite alphabets.Classes of memoryless sources defined by an envelope condition on the marginal distribution provide benchmarks for coding techniques originating from the theory of universal coding over finite alphabets. We prove general upperbounds on minimax regret and lower-bounds on minimax redundancy for such source classes. The general upper bounds emphasize the role of the Normalized Maximum Likelihood codes with respect to minimax regret in the infinite alphabet context. Lower bounds are derived by tailoring sharp bounds on the redundancy of Krichevsky-Trofimov coders for sources over finite alphabets. Up to logarithmic (resp. constant) factors the bounds are matching for source classes defined by algebraically declining (resp. exponentially vanishing) envelopes. Effective and (almost) adaptive coding techniques are described for the collection of source classes defined by algebraically vanishing envelopes. Those results extend our knowledge concerning universal coding to contexts where the key tools from parametric inference are known to fail. keywords: NML; countable alphabets; redundancy; adaptive compression; minimax; I. INTRODUCTIONThis paper is concerned with the problem of universal coding on a countably infinite alphabet X (say the set of positive integers N + or the set of integers N ) as described for example by .Throughout this paper, a source on the countable alphabet X is a probability distribution on the set X N of infinite sequences of symbols from X (this set is endowed with the σ-algebra generated by sets of the form n i=1 {x i }×X N where all x i ∈ X and n ∈ N). The symbol Λ will be used to denote various classes of sources on the countably infinite alphabet X . The sequence of symbols emitted by a source is denoted by the X AE -valued random variable X = (X n ) n∈N . If P denotes the distribution of X, P n denotes the distribution of X 1:n = X 1 , ..., X n , and we let Λ n = {P n : P ∈ Λ}. For any countable set X , let M 1 (X ) be the set of all probability measures on X .From Shannon noiseless coding Theorem (see Cover and Thomas, 1991), the binary entropy of P n , H(X 1:n ) = E P n [− log P (X 1:n )] provides a tight lower bound on the expected number of binary symbols needed to encode outcomes of P n . Throughout the paper, logarithms are in base 2. In the following, we shall only consider finite entropy sources on countable alphabets, and we implicitly assume that H(X 1:n ) < ∞. The expected redundancy of any distribution Q n ∈ M 1 (X n ), defined as the difference between the expected code length E P [− log Q n (X 1:n )] and H(X 1:n ), is equal to the Kullback-Leibler divergence (or relative entropy) D(P n , Q n ) = x∈X n P n {x} log P n (x) Q n (x) = P n log P n (X1:n) Q n (X1:n) . Universal coding attempts to develop sequences of coding probabilities (Q n ) n so as to minimize expected redundancy over a whole class of sources. Technically speaking, several distinct notions of universa...

show abstract

Section: A Pattern Codingmentioning

confidence: 99%

Coding on Countably Infinite Alphabets

Boucheron

Garivier

Gassiat

2009

IEEE Trans. Inform. Theory

Self Cite

111

View full text Add to dashboard Cite

show abstract

“…For example, the pattern of "banana" is 123232 and its dictionary is 1 → b, 2 → a, and 3 → n. Letting ∆ n ψ denote the collection of all pattern distributions, induced on sequences of length n by all i.i.d. distributions over any alphabet, a sequence of papers , Shamir [2006Shamir [ , 2004, Garivier [2009], , Acharya et al [2012Acharya et al [ , 2013 showed that although patterns carry essentially all the entropy, they can be compressed with redundancy 0.3 · n 1/3 ≤R(∆ n ψ ) ≤R(∆ n ψ ) ≤ n 1/3 · log 4 n as n → ∞. Namely, pattern redundancy too is sublinear in the block length and most significantly, is uniformly upper bounded regardless of the alphabet size (which can be even infinite).…”

Section: Previous Resultsmentioning

confidence: 99%

Section: Previous Resultsmentioning

confidence: 99%

Universal compression of power-law distributions

Falahatgar

Jafarpour

Orlitsky

et al. 2015

2015 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

English words and the outputs of many other natural processes are well-known to follow a Zipf distribution. Yet this thoroughly-established property has never been shown to help compress or predict these important processes. We show that the expected redundancy of Zipf distributions of order α > 1 is roughly the 1/α power of the expected redundancy of unrestricted distributions. Hence for these orders, Zipf distributions can be better compressed and predicted than was previously known. Unlike the expected case, we show that worstcase redundancy is roughly the same for Zipf and for unrestricted distributions. Hence Zipf distributions have significantly different worst-case and expected redundancies, making them the first natural distribution class shown to have such a difference.

show abstract

“…Unlike the mostly finite-alphabet results referred to so far, this paper is concerned with adaptive coding over a countably infinite alphabet X (say the set of positive integers N + or the set of integers N) as described for example in Kieffer (1978); Gyorfi et al (1993); Foster et al (2002); Orlitsky and Santhanam (2004); Ryabko et al (2008); Boucheron et al (2009); Garivier (2009); Bontemps (2011); Gassiat (2014); Bontemps et al (2014). This does not preclude the finite-alphabet case, which becomes a special instance.…”

Section: Contributions and Organization Of The Papermentioning

confidence: 99%

About Adaptive Coding on Countable Alphabets: Max-Stable Envelope Classes

Boucheron

Gassiat

Ohannessian

2015

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

In this paper, we study the problem of lossless universal source coding for stationary memoryless sources on countably infinite alphabets. This task is generally not achievable without restricting the class of sources over which universality is desired. Building on our prior work, we propose natural families of sources characterized by a common dominating envelope. We particularly emphasize the notion of adaptivity, which is the ability to perform as well as an oracle knowing the envelope, without actually knowing it. This is closely related to the notion of hierarchical universal source coding, but with the important difference that families of envelope classes are not discretely indexed and not necessarily nested.Our contribution is to extend the classes of envelopes over which adaptive universal source coding is possible, namely by including max-stable (heavy-tailed) envelopes which are excellent models in many applications, such as natural language modeling. We derive a minimax lower bound on the redundancy of any code on such envelope classes, including an oracle that knows the envelope. We then propose a constructive code that does not use knowledge of the envelope. The code is computationally efficient and is structured to use an Expanding Threshold for Auto-Censoring, and we therefore dub it the ETAC-code. We prove that the ETAC-code achieves the lower bound on the minimax redundancy within a factor logarithmic in the sequence length, and can be therefore qualified as a near-adaptive code over families of heavy-tailed envelopes. For finite and lighttailed envelopes the penalty is even less, and the same code follows closely previous results that explicitly made the lighttailed assumption. Our technical results are founded on methods from regular variation theory and concentration of measure.

show abstract

A Lower-Bound for the Maximin Redundancy in Pattern Coding

Cited by 9 publications

References 22 publications

Coding on Countably Infinite Alphabets

Coding on Countably Infinite Alphabets

Universal compression of power-law distributions

About Adaptive Coding on Countable Alphabets: Max-Stable Envelope Classes

Contact Info

Product

Resources

About