Source Coding for Quasiarithmetic Penalties

Abstract

Huffman coding finds a prefix code that minimizes mean codeword length for a given probability distribution over a finite number of items. Campbell generalized the Huffman problem to a family of problems in which the goal is to minimize not mean codeword length i pili but rather a generalized mean of the form ϕ −1 ( i piϕ(li)),where li denotes the length of the ith codeword, pi denotes the corresponding probability, and ϕ is a monotonically increasing cost function. Such generalized means -also known as quasiarithmetic or quasilinear means -have a number of diverse applications, including applications in queueing. Several quasiarithmetic-mean problems have novel simple redundancy bounds in terms of a generalized entropy. A related property involves the existence of optimal codes: For "well-behaved" cost functions, optimal codes always exist for (possibly infinite-alphabet) sources having finite generalized entropy. Solving finite instances of such problems is done by generalizing an algorithm for finding length-limited binary codes to a new algorithm for finding optimal binary codes for any quasiarithmetic mean with a convex cost function. This algorithm can be performed using quadratic time and linear space, and can be extended to other penalty functions, some of which are solvable with similar space and time complexity, and others of which are solvable with slightly greater complexity. This reduces the computational complexity of a problem involving minimum delay in a queue, allows combinations of previously considered problems to be optimized, and greatly expands the space of problems solvable in quadratic time and linear space. The algorithm can be extended for purposes such as breaking ties among possibly different optimal codes, as with bottom-merge Huffman coding. Index TermsOptimal prefix code, Huffman algorithm, generalized entropies, generalized means, quasiarithmetic means, queueing.

Optimal Prefix Codes for Infinite Alphabets With Nonlinear Costs

“…If a > 0.5, a code with finite penalty exists if and only if Rényi entropy of order α(a) = (1 + log 2 a) −1 is finite, as shown in [16]. It was Campbell who first noted the connection between the optimal code's penalty, L a (P, N * ), and Rényi entropy…”

“…One might ask whether there must exist an optimal code or if there can be an infinite sequence of codes of decreasing penalty without any code achieving the limit penalty value. Fortunately the answer is the former, the proof being a special case of Theorem 2 in [16] (a generalization of the result for the expected-length penalty [17]). The question is then how to find one of these optimal source codes given parameter a and probability measure P .…”

D-ary Bounded-Length Huffman Coding

“…1) It can be nonbinary, a case considered by Huffman in his original paper [4]; 2) There is a maximum codeword length, a restriction previously considered, e.g., [9] in O(n 3 l max log D) time [10] and O(n 2 log D) space, but solved efficiently only for binary coding, e.g., [8] in O(nl max ) time O(n) space and most efficiently in [11]; 3) There is a minimum codeword length, a novel restriction; 4) The penalty can be nonlinear, a modification previously considered, but only for binary coding, e.g., [12]. The minimum size constraint on codeword length requires a relatively simple change of solution range to [8].…”

“…Before presenting an algorithm for optimizing the above problem, we introduce a notation for codes that generalizes one first presented in [6] and modified in [12].…”

“…The optimal solution, a function of total width ρ tot and items I, is denoted CC(I, ρ tot ) (the optimal coin collection for I and ρ tot ). Note that, due to ties, this need not be a unique solution, but the algorithm proposed here is deterministic; that is, it finds one specific solution, much like bottom-merge Huffman coding [13] or the corresponding length-limited problem [12], [14] Because…”