A coding theorem and Rényi's entropy

Campbell, L. L.

doi:10.1016/s0019-9958(65)90332-3

Cited by 254 publications

(272 citation statements)

References 2 publications

Supporting

Mentioning

266

Contrasting

Order By: Relevance

“…But this is not always the case. Campbell [17] introduced the mean codeword length which implies that cost is an exponential function of code length. The cost of encoding the source is expressed by the exponential average…”

Section: Source Coding With Campbell Measure Of Lengthmentioning

confidence: 99%

An Algorithm to Generate Probabilities with Specified Entropy

Parkash¹,

Kakkar²

2015

View full text Add to dashboard Cite

The present communication offers a method to determine an unknown discrete probability distribution with specified Tsallis entropy close to uniform distribution. The relative error of the distribution obtained has been compared with the distribution obtained with the help of mathematica software. The applications of the proposed algorithm with respect to Tsallis source coding, Huffman coding and cross entropy optimization principles have been provided.

show abstract

Section: Source Coding With Campbell Measure Of Lengthmentioning

confidence: 99%

An Algorithm to Generate Probabilities with Specified Entropy

Parkash¹,

Kakkar²

2015

View full text Add to dashboard Cite

show abstract

“…Other forms of lengths have also been considered, the first and fundamental contribution being Campbell's one [11]. In Shannon's result, low probabilities yield very long words.…”

Section: Source Codingmentioning

confidence: 99%

“…The remarkable result [11] is that just as Shannon entropy is the lower bound on the average codeword length of an uniquely decodable code, the Rényi entropy of order q, with q = 1/(β + 1), is the lower bound on the exponentially weighted codeword length, see line 2 of Table 1.…”

Section: Source Codingmentioning

confidence: 99%

On escort distributions, q-gaussians and Fisher information

Mohammad‐Djafari¹,

Bercher

Bessìère³

2011

AIP Conference Proceedings

View full text Add to dashboard Cite

Abstract. Escort distributions are a simple one parameter deformation of an original distribution p. In Tsallis extended thermostatistics, the escort-averages, defined with respect to an escort distribution, have revealed useful in order to obtain analytical results and variational equations, with in particular the equilibrium distributions obtained as maxima of Rényi-Tsallis entropy subject to constraints in the form of a q-average. A central example is the q-gaussian, which is a generalization of the standard gaussian distribution.In this contribution, we show that escort distributions emerge naturally as a maximum entropy trade-off between the distribution p(x) and the uniform distribution. This setting may typically describe a phase transition between two states. But escort distributions also appear in the fields of multifractal analysis, quantization and coding with interesting consequences. For the problem of coding, we recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy and exhibit the links with escort distributions together with pratical implications.That q-gaussians arise from the maximization of Rényi-Tsallis entropy subject to a q-variance constraint is a known fact. We show here that the (squared) q-gaussian also appear as a minimum of Fisher information subject to the same q-variance constraint.

show abstract

“…Campbell (cf. [6]) introduced a generalized measure for code length and has shown that this measure is related to Rényi's entropy. This strongly encourages to work with Rényi-α-entropy as a generalized measure of complexity for the quantizers.…”

Section: Introduction and Basic Notationmentioning

confidence: 99%

“…The author is indebted to the referee for pointing out some errors and simplifications in an earlier version of this paper as well as for the communication of reference [6].…”

mentioning

confidence: 99%

Error bounds for high-resolution quantization with Rényi-α-entropy constraints

Kreitmeier

2010

Acta Math Hung

View full text Add to dashboard Cite

We consider the problem of optimal quantization with norm exponent r > 0 for Borel probability measures on R d under constrained Rényi-α-entropy of the quantizers. If the bound on the entropy becomes large, then sharp asymptotics for the optimal quantization error are well-known in the special cases α = 0 (memory-constrained quantization) and α = 1 (Shannon-entropy-constrained quantization). In this paper we determine sharp asymptotics for the optimal quantization error under large entropy bound with entropy parameter α ∈ [1 + r/d, ∞]. For α ∈ [0, 1 + r/d[ we specify the asymptotical order of the optimal quantization error under large entropy bound. The optimal quantization error is decreasing exponentially fast with the entropy bound and the exact rate is determined for all α ∈ [0, ∞].

show abstract

A coding theorem and Rényi's entropy

Cited by 254 publications

References 2 publications

An Algorithm to Generate Probabilities with Specified Entropy

An Algorithm to Generate Probabilities with Specified Entropy

On escort distributions, q-gaussians and Fisher information

Error bounds for high-resolution quantization with Rényi-α-entropy constraints

Contact Info

Product

Resources

About