Tight bounds for universal compression of large alphabets

Acharya, Jayadev; Das, Hirakendu; Jafarpour, Ashkan; Orlitsky, Alon; Suresh, Ananda Theertha

doi:10.1109/isit.2013.6620751

Cited by 7 publications

(14 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the pattern of "banana" is 123232 and its dictionary is 1 → b, 2 → a, and 3 → n. Letting ∆ n ψ denote the collection of all pattern distributions, induced on sequences of length n by all i.i.d. distributions over any alphabet, a sequence of papers , Shamir [2006Shamir [ , 2004, Garivier [2009], , Acharya et al [2012Acharya et al [ , 2013 showed that although patterns carry essentially all the entropy, they can be compressed with redundancy 0.3 · n 1/3 ≤R(∆ n ψ ) ≤R(∆ n ψ ) ≤ n 1/3 · log 4 n as n → ∞. Namely, pattern redundancy too is sublinear in the block length and most significantly, is uniformly upper bounded regardless of the alphabet size (which can be even infinite).…”

Section: Previous Resultsmentioning

confidence: 99%

Section: Previous Resultsmentioning

confidence: 99%

“…Poisson sampling: In the standard sampling method, where a distribution is sampled n times, the multiplicities are dependent, for example they add up to n. Hence, calculating redundancy under this sampling often requires various concentration inequalities, complicating the proofs. A useful approach to make them independent and hence simplify the analysis is to sample the distribution n ′ times, where n ′ is a Poisson random variable with mean n. Often called as Poisson sampling, this approach has been used in universal compression to simplify the analysis Acharya et al [2012Acharya et al [ , 2013, Yang and Barron [2013], Acharya et al [2014b].…”

Section: Lower Boundmentioning

confidence: 99%

See 2 more Smart Citations

Universal compression of power-law distributions

Falahatgar

Jafarpour

Orlitsky

et al. 2015

2015 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

English words and the outputs of many other natural processes are well-known to follow a Zipf distribution. Yet this thoroughly-established property has never been shown to help compress or predict these important processes. We show that the expected redundancy of Zipf distributions of order α > 1 is roughly the 1/α power of the expected redundancy of unrestricted distributions. Hence for these orders, Zipf distributions can be better compressed and predicted than was previously known. Unlike the expected case, we show that worstcase redundancy is roughly the same for Zipf and for unrestricted distributions. Hence Zipf distributions have significantly different worst-case and expected redundancies, making them the first natural distribution class shown to have such a difference.

show abstract

Section: Previous Resultsmentioning

confidence: 99%

Section: Previous Resultsmentioning

confidence: 99%

Section: Lower Boundmentioning

confidence: 99%

See 1 more Smart Citation

Universal compression of power-law distributions

Falahatgar

Jafarpour

Orlitsky

et al. 2015

2015 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Poisson sampling provides simplified analysis in various machine learning and statistical problems [15]. To the best of our knowledge, it was first used for universal compression in [2], and [1] to prove optimal bounds on pattern redundancy. Recently, [25] also apply Poisson sampling to provide simpler coding schemes for i.i.d.…”

Section: New Results and Techniquesmentioning

confidence: 99%

“…Since the worst case redundancy is always larger than the average redundancy, a class with infinite average redundancy has infinite worst case redundancy. However, there exist classes of distributions with a finite average redundancy and infinite worst case redundancy (See Example 1 in [1]).…”

Section: A Note On Expected Redundancymentioning

confidence: 99%

Universal Compression of Envelope Classes: Tight Characterization via Poisson Sampling

Acharya,

Jafarpour,

Orlitsky

et al. 2014

Preprint

Self Cite

View full text Add to dashboard Cite

The Poisson-sampling technique eliminates dependencies among symbol appearances in a random sequence. It has been used to simplify the analysis and strengthen the performance guarantees of randomized algorithms. Applying this method to universal compression, we relate the redundancies of fixed-length and Poisson-sampled sequences, use the relation to derive a simple single-letter formula that approximates the redundancy of any envelope class to within an additive logarithmic term. As a first application, we consider i.i.d. distributions over a small alphabet as a step-envelope class, and provide a short proof that determines the redundancy of discrete distributions over a small alphabet up to the first order terms. We then show the strength of our method by applying the formula to tighten the existing bounds on the redundancy of exponential and power-law classes, in particular answering a question posed by Boucheron, Garivier and Gassiat [6].

show abstract

On near‐redundancy and identifiability of parametric hazard regression models under censoring

Rubio

Espíndola²,

Montoya³

2023

Biometrical J

View full text Add to dashboard Cite

We study parametric inference on a rich class of hazard regression models in the presence of right‐censoring. Previous literature has reported some inferential challenges, such as multimodal or flat likelihood surfaces, in this class of models for some particular data sets. We formalize the study of these inferential problems by linking them to the concepts of near‐redundancy and practical nonidentifiability of parameters. We show that the maximum likelihood estimators of the parameters in this class of models are consistent and asymptotically normal. Thus, the inferential problems in this class of models are related to the finite‐sample scenario, where it is difficult to distinguish between the fitted model and a nested nonidentifiable (i.e., parameter‐redundant) model. We propose a method for detecting near‐redundancy, based on distances between probability distributions. We also employ methods used in other areas for detecting practical nonidentifiability and near‐redundancy, including the inspection of the profile likelihood function and the Hessian method. For cases where inferential problems are detected, we discuss alternatives such as using model selection tools to identify simpler models that do not exhibit these inferential problems, increasing the sample size, or extending the follow‐up time. We illustrate the performance of the proposed methods through a simulation study. Our simulation study reveals a link between the presence of near‐redundancy and practical nonidentifiability. Two illustrative applications using real data, with and without inferential problems, are presented.

show abstract

Tight bounds for universal compression of large alphabets

Cited by 7 publications

References 22 publications

Universal compression of power-law distributions

Universal compression of power-law distributions

Universal Compression of Envelope Classes: Tight Characterization via Poisson Sampling

On near‐redundancy and identifiability of parametric hazard regression models under censoring

Contact Info

Product

Resources

About