An entropy estimator improving mean squared error

Abstract

SUMMARYEntropy estimation for a memory-less information source has mainly been performed by the following process under the condition that the occurrence probability of each source symbol is unknown: First, the occurrence probabilities of source symbols are estimated; then the entropy is estimated by substituting the estimated occurrence probabilities into the entropy function. Such an entropy estimation is not optimum in the sense of least squares error; it causes a large error, particularly for small sample size. In this paper, we propose an entropy estimator that strictly minimizes the mean squared error for a two-ary memoryless information source; it is formulated as a function of the number of one type of source symbols contained in the sample set. Furthermore, we propose a technique for estimating entropy of an arbitrary M-ary memory-less information source by recursively applying the proposed entropy estimator for a two-ary memory-less information source. Numerical experiments show that this technique improves mean squared error compared to the conventional method even though the guarantee that it is the least squares error estimator has been lost.

An approximate method for Bayesian entropy estimation for a discrete random variable

“…Such a Bayesian entropy estimator has already been proposed for a random variable that takes only two sorts of values [6], [7]. However, a Bayesian entropy estimator for a random variable that takes numerous sorts of values requires large amount of calculation cost generally.…”