Rényi to Rényi ¿ Source Coding under Siege

Abstract

A novel lossless source coding paradigm applies to problems of unreliable lossless channels with low bit rates, in which a vital message needs to be transmitted prior to termination of communications. This paradigm can be applied to Alfréd Rényi's secondhand account of an ancient siege in which a spy was sent to scout the enemy but was captured. After escaping, the spy returned to his base in no condition to speak and unable to write. His commander asked him questions that he could answer by nodding or shaking his head, and the fortress was defended with this information. Rényi told this story with reference to prefix coding, but maximizing probability of survival in the siege scenario is distinct from yet related to the traditional source coding objective of minimizing expected codeword length. Rather than finding a code minimizing expected codeword length P n i=1 p(i)l(i), the siege problem involves maximizing P n i=1 p(i)θ l(i) for a known θ ∈ (0, 1). When there are no restrictions on codewords, this problem can be solved using a known generalization of Huffman coding. The optimal solution has coding bounds which are functions of Rényi entropy; in addition to known bounds, new bounds are derived here. The alphabetically constrained version of this problem has applications in search trees and diagnostic testing. A novel dynamic programming algorithm -based upon the oldest known algorithm for the traditional alphabetic problem -optimizes this problem in O(n 3 ) time and O(n 2 ) space, whereas two novel approximation algorithms can find a suboptimal solution faster: one in linear time, the other in O(n log n). Coding bounds for the alphabetic version of this problem are also presented.

Alphabetic coding with exponential costs

“…With (2), this no longer holds. Consider a = 0.6 with input weights w = (8, 1,9,6). The splitting point of (8, 1,9) is s = 3 (w(s) = w(3) = 9, yielding subtrees with (8, 1) and (9)), and the splitting point of (1, 9, 6) is s = 4 (w(s) = 6).…”

“…The splitting point of (8, 1,9) is s = 3 (w(s) = w(3) = 9, yielding subtrees with (8, 1) and (9)), and the splitting point of (1, 9, 6) is s = 4 (w(s) = 6). However, the optimal splitting point of (8, 1,9,6) …”

“…Continuing on, we obtain a tree that is not necessarily alphabetical, but which has the same lengths as an alphabetic tree which can be easily reconstructed (optimally) solving the problem (for a > 1). However, consider again a = 0.6, this time for weights (8,1,9,6,2). The Hu-Kleitman-Tamaki method first combines 6 and 2, then 8 and 1, then the first combined node with 9, and finally the remaining two nodes, resulting in a tree with lengths l = (2, 2, 2, 3, 3) and L a (w, l ) ≈ −4.121.…”

“…Similarly, although the nonalphabetic terms here are useful in combination with other known bounds (e.g., [1]) to further improve these alphabetic bounds, we omit the more complicated improved bounds. …”

“…,n} of the tree as l(i), and the weight (usually probability) of the leaf as w(i). Given a set of weights, Huffman's algorithm [7] finds a tree minimizing cost function n i=1 w(i)l(i) (1) and Hu and Tucker's algorithm [6] finds an optimal alphabetic tree:…”