Tight Bounds on the Redundancy of Huffman Codes

Abstract

In this paper we study the redundancy of Huffman codes. In particular, we consider sources for which the probability of one of the source symbols is known. We prove a conjecture of Ye and Yeung regarding the upper bound on the redundancy of such Huffman codes, which yields in a tight upper bound. We also derive a tight lower bound for the redundancy under the same assumption. We further apply the method introduced in this paper to other related problems. It is shown that several other previously known bounds with different constraints follow immediately from our results.Comment: 23 pages, 7 figures, accepted for publication in IEEE Transaction on Information Theor

smallWig: parallel compression of RNA-seq WIG files

“…There exist a number of results on the upper and lower bounds of the expected redundancy r ¼ E X ðrðXÞÞ for a random variable X. For example, Gallager (1978) showed an upper bound based on the largest symbol probability; Capocelli and De Santis (1991) bounded the redundancy both from above and below based on the largest and the smallest symbol probability and Mohajer et al (2012) showed a tight upper and lower bound based on one known symbol probability p. In particular, in the latter case, the redundancy is lower bounded by r ! mp À HðpÞ À ð1 À pÞlog 2 ð1 À 2 Àm Þ; where m > 0 is either dÀlog 2 peorbÀlog 2 pc, depending on which value minimizes the overall expression.…”

On the redundancy of Huffman codes with exponential objectives

“…(9) Proof: 1) Lower bound: The lower bound calculation is: , where π d 0 as the first root of the equality of the two terms in the maximization at (9). The tight upper bounds for d < ∞ are approached by p = (p j , 1 − p j − ǫ, ǫ).…”

“…For both the traditional and exponential utilities, we can improve on the unit-sized bound given the probability of one of the source events. This was first done with the constraint that the given probability be the most probable of these events [7], but here, as in some subsequent work [4], [8], [9], we drop this constraint. Without loss of generality, we call the source symbols {1, 2, .…”

“…In traditional linear optimization, upper and lower bounds for R d are known such that probability distributions can be found achieving or approaching these bounds [8], [9]; i.e., they are tight. In the exponential cases, [4] took a ↑ ∞ (d ↑ ∞) and a ↓ 1 (d ↓ 0), using inequality relations to find not-necessarilytight bounds on these problems in terms of tight bounds for the limit cases.…”

Entropy and Average Cost of AUH Codes

“…These sources were extensively analysed, concerning bounds on average codeword length, entropy and redundancy for different types of probability distribution [7]- [9]. The AUH sources appear in a wide variety of situations in the real world, because this class of sources have the property of achieving minimum redundancy in different situations and minimal average cost in highly unbalanced cost regime [10]- [12]. These properties determine a wide range of applications and motivate us to study these sources from an information theoretic perspective.…”