Confidence intervals for the mutual information

Abstract

THIS PAPER IS ELIGIBLE FOR THE STUDENT PAPER AWARD"By combining a bound on the absolute value of the difference of mutual information between two joint probability distributions with a fixed variational distance, and a bound on the probability of a maximal deviation in variational distance between a true joint probability distribution and an empirical joint probability distribution, confidence intervals for the mutual information of two random variables with finite alphabets are established. Different from previous results, these intervals do not need any assumptions on the distribution and the sample size.

A tight lower bound on the mutual information of a binary and an arbitrary finite random variable as a function of the variational distance

“…Together with the bound on the probability of a maximal variational distance between the true joint distribution and an empirical joint distribution (see [6], and especially an refinement of it which drops the dependence on the true distribution [4, Lemma 3]) the given bound can be used to construct a reasonably tight lower bound of the confidence interval for mutual information. Such an application can be found in [8]. In mutual information estimation with confidence intervals, the bound given is especially useful, when the marginal probability distribuition is far from being uniform.…”