In 1977, Keane and Smorodinsky [9] showed that there exists a finitary homomorphism from any finite-alphabet Bernoulli process to any other finite-alphabet Bernoulli process of strictly lower entropy. In 1996, Serafin [17] proved the existence of a finitary homomorphism with finite expected coding length. In this paper, we construct such a homomorphism in which the coding length has exponential tails. Our construction is source-universal, in the sense that it does not use any information on the source distribution other than the alphabet size and a bound on the entropy gap between the source and target distributions. We also indicate how our methods can be extended to prove a source-specific version of the result for Markov chains.
Abstract-We give a method for estimating the empirical Shannon entropy of a distribution in the streaming model of computation. Our approach reduces this problem to the wellstudied problem of estimating frequency moments. The analysis of our approach is based on new results which establish quantitative bounds on the rate of convergence of Rényi entropy towards Shannon entropy.
Let p and q be probability vectors with the same entropy h. Denote by B(p) the Bernoulli shift indexed by ℤ with marginal distribution p. Suppose that φ is a measure-preserving homomorphism from B(p) to B(q). We prove that if the coding length of φ has a finite 1/2 moment, then σ2p=σ2q, where σ2p=∑ ipi(−log pi−h)2 is the informational variance of p. In this result, the 1/2 moment cannot be replaced by a lower moment. On the other hand, for any θ<1, we exhibit probability vectors p and q that are not permutations of each other, such that there exists a finitary isomorphism Φ from B(p) to B(q) where the coding lengths of Φ and of its inverse have a finite θ moment. We also present an extension to ergodic Markov chains.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.