Non–Parametric Estimation of Mutual Information through the Entropy of the Linkage

Abstract

A new, non-parametric and binless estimator for the mutual information of a d-dimensional random vector is proposed. First of all, an equation that links the mutual information to the entropy of a suitable random vector with uniformly distributed components is deduced. When d = 2 this equation reduces to the well known connection between mutual information and entropy of the copula function associated to the original random variables. Hence, the problem of estimating the mutual information of the original random vector is reduced to the estimation of the entropy of a random vector obtained through a multidimensional transformation. The estimator we propose is a two-step method: first estimate the transformation and obtain the transformed sample, then estimate its entropy. The properties of the new estimator are discussed through simulation examples and its performances are compared to those of the best estimators in the literature. The precision of the estimator converges to values of the same order of magnitude of the best estimator tested. However, the new estimator is unbiased even for larger dimensions and smaller sample sizes, while the other tested estimators show a bias in these cases.

Inference of Causal Information Flow in Collective Animal Behavior

“…For fixed value of k, let (i) denotes the distance from w i ≡ (x i , y i , z i ) to its kth nearest neighbor, where distance is measured as the max-norm in the joint space, w i −w j xyz = max{ x i −x j x , y i −y j y , z i −z j z }, and the norms used in the subspaces can be arbitrary but often time the max norm is used (which is our choice for this paper) as well. From this we obtain • n xz (i): number of points (x j , z j ) ( j = i) with (x j , z j ) − (x i , z i ) xz = max{ x j − x i x , z j − z i z } < (i) • n yz (i): number of points (y j , z j ) ( j = i) with (y j , z j ) − (y i , z i ) yz = max{ y j − y i y , z j − z i z } < (i) • n z (i): number of points z j ( j = i) with z j − z i z < (i) Several recent papers have focused on reducing the finitesample bias of the KSG type of estimators [70]- [72] or developing other types of estimators [73].…”

A Copula-Based Method to Build Diffusion Models with Prescribed Marginal and Serial Dependence

“…After applying equation (11) again to get c X s,t F X s (z), F X t (x) from the transition of X, equation (22) is easily reorganized into equation (19).…”

“…[22,25,34]). They have found application in many different fields ranging from finance and insurance [12,15], to reliability [1,33], stochastic ordering [36], geophysics [42], neuroscience [2,3,20,23,35,41], statistics [19] and many more.…”

Estimating Differential Entropy using Recursive Copula Splitting

“…Substituting Equation ( 2 ) into Equation ( 1 ) yields, where is the entropy of the k ’th marginal, to be computed using appropriate 1D estimators, and is the entropy of the copula. Using Sklar’s theorem has been previously suggested as a method for calculating the mutual information between variables, which is identical to the copula entropy [ 5 , 28 , 29 , 30 ]. The new approach here is in showing that can be efficiently estimated recursively, similar to the k DP approach.…”