A tight lower bound on the mutual information of a binary and an arbitrary finite random variable as a function of the variational distance

Abstract

THIS PAPER IS ELIGIBLE FOR THE STUDENT PAPER AWARD".In this paper a numerical method is presented, which finds a lower bound for the mutual information between a binary and an arbitrary finite random variable with joint distributions that have a variational distance not greater than a known value to a known joint distribution. This lower bound can be applied to mutual information estimation with confidence intervals.

Confidence intervals for the mutual information

“…In this section the different possibilities for the construction of the confidence intervals, which just have been discussed are compared in two numerical examples. In these particular examples it can be seen that the lower bound conjectured in [4] (called Method 1) matches the lower bound of preprint [7] (called Method 2) which gives a further indication for the correctness of at least the lower bound in [4] (though there is still no proof available).…”

“…( 1) was used. In (7) the bound of Ho and Yeung [3,Theorem 6] was applied together with the assumption M x ≤ M y and therefore, by the assumption…”

“…Besides this in the preprint [7], an algorithm for finding the lower bound of the confidence interval for a binary and an arbitrary finite random variable is given. This bound is tight in terms of the maximal variational distance between the empirical and the true joint distribution.…”