Critical Values of a Kernel Density-based Mutual Information Estimator

Abstract

Abstract-Recently, mutual information (MI) has become widely recognized as a statistical measure of dependence that is suitable for applications where data are non-Gaussian, or where the dependency between variables is non-linear. However, a significant disadvantage of this measure is the inability to define an analytical expression for the distribution of MI estimators, which are based upon a finite dataset. This paper deals specifically with a popular kernel density based estimator, for which the distribution is determined empirically using Monte Carlo simulation. The application of the critical values of MI derived from this distribution to a test for independence is demonstrated within the context of a benchmark input variable selection problem.

Applying information-based methods in importance–performance analysis when the information of importance is unavailable

“…A higher mutual information value is a result of strong correlation, whereas a mutual information value of zero indicates uncorrelated variables (May et al 2006). Specifically, the mutual information M I (X, Y ) between random variables X and Y measures the amount of information in X that can be predicted when Y is known.…”

“…In addition, mutual information method can be applied to evaluate the dependence between two random variables and measures how much the entropy of one random variable is reduced by knowledge of another. Therefore, mutual information method is an ideal measure of stochastic dependence between two random variables (May et al 2006).…”