Definition of entropy by means of a coding problem

Campbell, L. L.

doi:10.1007/bf00537132

Cited by 58 publications

(55 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…which should not be surprising given the similar relationship between Huffman-optimal codes and Shannon entropy [24], which corresponds to a → 1 (α → 1) [8], [25]; due to this correspondence, Shannon entropy is sometimes expressed as H 1 (P ).…”

Section: Background: Finite Alphabetsmentioning

confidence: 99%

See 1 more Smart Citation

Optimal Prefix Codes for Infinite Alphabets With Nonlinear Costs

Baer¹

2008

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-Let P = {p(i)} be a measure of strictly positive probabilities on the set of nonnegative integers. Although the countable number of inputs prevents usage of the Huffman algorithm, there are nontrivial P for which known methods find a source code that is optimal in the sense of minimizing expected codeword length. For some applications, however, a source code should instead minimize one of a family of nonlinear objective functions, β-exponential means, those of the form log a P i p(i)a n(i) , where n(i) is the length of the ith codeword and a is a positive constant. Applications of such minimizations include a novel problem of maximizing the chance of message receipt in single-shot communications (a < 1) and a previously known problem of minimizing the chance of buffer overflow in a queueing system (a > 1). This paper introduces methods for finding codes optimal for such exponential means. One method applies to geometric distributions, while another applies to distributions with lighter tails. The latter algorithm is applied to Poisson distributions and both are extended to alphabetic codes, as well as to minimizing maximum pointwise redundancy. The aforementioned application of minimizing the chance of buffer overflow is also considered.

show abstract

Section: Background: Finite Alphabetsmentioning

confidence: 99%

“…Other utility functions, however, have been considered for finitealphabet codes. Campbell [8] introduced a problem in which the penalty to minimize, given some continuous (strictly) monotonic increasing cost function ϕ(x) : R + → R + , is L(P, N, ϕ) = ϕ and specifically considered the exponential subcases with exponent a > 1:…”

mentioning

confidence: 99%

Optimal Prefix Codes for Infinite Alphabets With Nonlinear Costs

Baer¹

2008

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…This was previously proposed by Campbell [4] and algorithmically solved as an extension of Huffman's algorithm (and thus with linear time and space for sorted probability inputs) in [3], [7], [10], [11]. As previously indicated, in [3], [7] this is a step in minimizing the probability of buffer overflow in a queueing system.…”

Section: Examplesmentioning

confidence: 99%

“…where here infimum is used because this definition applies to codes with infinite, as well as finite, input alphabets [4]. …”

Section: A Bounds and The Subtranslatory Propertymentioning

confidence: 99%

See 1 more Smart Citation

Source Coding for Quasiarithmetic Penalties

Baer

2006

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Huffman coding finds a prefix code that minimizes mean codeword length for a given probability distribution over a finite number of items. Campbell generalized the Huffman problem to a family of problems in which the goal is to minimize not mean codeword length i pili but rather a generalized mean of the form ϕ −1 ( i piϕ(li)),where li denotes the length of the ith codeword, pi denotes the corresponding probability, and ϕ is a monotonically increasing cost function. Such generalized means -also known as quasiarithmetic or quasilinear means -have a number of diverse applications, including applications in queueing. Several quasiarithmetic-mean problems have novel simple redundancy bounds in terms of a generalized entropy. A related property involves the existence of optimal codes: For "well-behaved" cost functions, optimal codes always exist for (possibly infinite-alphabet) sources having finite generalized entropy. Solving finite instances of such problems is done by generalizing an algorithm for finding length-limited binary codes to a new algorithm for finding optimal binary codes for any quasiarithmetic mean with a convex cost function. This algorithm can be performed using quadratic time and linear space, and can be extended to other penalty functions, some of which are solvable with similar space and time complexity, and others of which are solvable with slightly greater complexity. This reduces the computational complexity of a problem involving minimum delay in a queue, allows combinations of previously considered problems to be optimized, and greatly expands the space of problems solvable in quadratic time and linear space. The algorithm can be extended for purposes such as breaking ties among possibly different optimal codes, as with bottom-merge Huffman coding. Index TermsOptimal prefix code, Huffman algorithm, generalized entropies, generalized means, quasiarithmetic means, queueing.

show abstract