Nonproduct Data-Dependent Partitions for Mutual Information Estimation: Strong Consistency and Applications

A near-optimal (minimax) tree-structured partition for mutual information estimation

“…This last result presented by the authors in a companion manuscript [12]. At this point, it is important to highlight the adaptation character of our TSP, which a function of the data finds the tree's topology that offers a near-optimal estimation-approximation error tradeoff (Theorem 2).…”

“…To conclude, considering the inequality in (12) and the distribution free bounds obtained for its RHS terms (in (13), (14) and (17), respectively), we obtain (8) ∀ ∈ (0, 3).…”

“…In addition, Π needs to satisfy a product bin condition, i.e., ∀z n 1 = (z 1 , .., z n ) ∈ R d·n every event A ∈ π n (z n 1 ) is expressed by [7], A = A 1 × A 2 , where A 1 ∈ B(R p ) and A 2 ∈ B(R q ). With this, the learning-estimation process involves three phases: first, to use the empirical data to partition R d by π n (Z n 1 ), second, to use again the data to estimate P X,Y and P X × P Y restricted to the sigma field σ(π n (Z n 1 )) 1 , and finally, to consider the plug-in technique to get an empirical MI estimate on (R d , σ(π n (Z n 1 ))) [12]. Concerning the phase 2, the product bin condition is needed to estimate P X,Y as well as the reference measure P X × P Y only based on the iid realizations of the joint distribution P X,Y [7], [12].…”

“…With this, the learning-estimation process involves three phases: first, to use the empirical data to partition R d by π n (Z n 1 ), second, to use again the data to estimate P X,Y and P X × P Y restricted to the sigma field σ(π n (Z n 1 )) 1 , and finally, to consider the plug-in technique to get an empirical MI estimate on (R d , σ(π n (Z n 1 ))) [12]. Concerning the phase 2, the product bin condition is needed to estimate P X,Y as well as the reference measure P X × P Y only based on the iid realizations of the joint distribution P X,Y [7], [12]. More precisely, let P denote the joint distribution and P n its empirical version, i.e., P n (A) = 1 n n i=1 I A (Z i ), ∀A ∈ B(R d ), hence, the MI estimate is given byÎ n (π n (Z n 1 )) =…”

“…This adaptive TSP estimate shows promising empirical evidence, although ensuring strong consistency remains an open problem [7]. Alternatively, Wang et al [8], [9] and more recently Silva et al [10], [11], [12] studied the role of a more general family of data-driven partitions, based on partition schemes [6], [13]. The work presented in this paper builds upon this formulation, where the learning and adaptation advantages of TSP schemes are further explored [14], [13], [15], [16].…”