Ensemble Estimation of Information Divergence †

This paper studies convergence of empirical measures smoothed by a Gaussian kernel. Specifically, consider approximating P * Nσ, for Nσ N (0, σ 2 I d ), byPn * Nσ, wherePn is the empirical measure, under different statistical distances. The convergence is examined in terms of the Wasserstein distance, total variation (TV), Kullback-Leibler (KL) divergence, and χ 2 -divergence. We show that the approximation error under the TV distance and 1-Wasserstein distance (W1) converges at the rate e O(d) n − 1 2 in remarkable contrast to a (typical) n − 1 d rate for unsmoothed W1(and d ≥ 3). Similarly, for the KL divergence, squared 2-Wasserstein distance (W 2 2 ), and χ 2 -divergence convergence rate is e O(d) n −1 , but only provided that P achieves finite input-output χ 2 mutual information across the additive white Gaussian noise (AWGN) channel. If the latter condition is not met, the rate changes to ω n −1 for the KL divergence and W 2 2 , while the χ 2 -divergence becomes infinite -a curious dichotomy. As a main application we consider estimating the differential entropy h(S + Z), where S ∼ P and Z ∼ Nσ are independent d-dimensional random variables. The distribution P is unknown and belongs to some nonparametric class, but n independently and identically distributed (i.i.d) samples from it are available. Despite the regularizing effect of noise, we first show that any good estimator (within an additive gap) for this problem must have a sample complexity that is exponential in d. We then leverage the empirical approximation results to show that the absolute-error risk of the plug-in estimator converges as e O(d) n − 1 2 , thus attaining the parametric rate. This establishes the plug-in estimator as minimax rate-optimal for the considered problem, with sharp dependence of the convergence rate both on n and on d. We provide numerical results comparing the performance of the plug-in estimator to that of general-purpose (unstructured) differential entropy estimators (based on kernel density estimation (KDE) or k nearest neighbors (kNN) techniques) applied to samples of S +Z. These results reveal a significant empirical superiority of the plug-in to stateof-the-art KDE and kNN methods. As a motivating utilization of the plug-in approach, we estimate information flows in deep neural networks and discuss Tishby's Information Bottleneck and the compression conjecture, among others.1 n n i=1 δ Si is the empirical measure. 1 Due to the popularity of the additive Gaussian noise model, we start by exploring this smoothed empirical approximation problem in detail, under several additional statistical distances. A. Convergence of Empirical Measures Smoothed by a Gaussian KernelConsider the empirical approximation error Eδ(P S n * N σ , P * N σ ) under some statistical distance δ. Various choices of δ are considered, such as the 1-Wasserstein and (squared) 2-Wasserstein distances, total variation (TV), Kullback-Leibler (KL) divergence, and χ 2 -divergence. We show that, when P is subgaussian, the approximation error under the 1-Was...

show abstract

“…Thus, with these parameters, a negligible bias requires n to be at least 2 0.99d . 8 The Q-function is defined as Q(x)…”

Section: Bias Lower Boundmentioning

confidence: 99%

Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation

Goldfeld

Greenewald

Niles-Weed

et al. 2020

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thus, we quantify dependence by D α (g,g), the Rényi divergence of g(y, z) with respect tog(y, z). This can also be used to define the Rényi mutual information [16,17] as:…”

Section: Entropy As Measure Of Dependencementioning

confidence: 99%

“…Noshad et al [26] circumvented the plug-in density and proposed a direct estimation method based on a graph-theoretical interpretation. Very recently, Moon et al [17] derived mean squared error convergence rates of kernel density-based plug-in estimators of mutual information and proposed ensemble estimators that achieve the parametric rate, although there are several restrictions on the densities and the kernel used.…”

Section: Estimator Of the Rényi Entropymentioning

confidence: 99%

“…In this section, we present a simulation study to compare the true value of the dependence measure with the measures obtained with MST and FMST estimators. A direct method to estimate the Rényi mutual information based on a kernel density estimator was given by Moon et al [17], which was used for comparison.…”

Section: Comparison With the Exact Value Of The Rényi Divergencementioning

confidence: 99%

“…We varied the value of ρ and studied the behavior of the MST and FMST estimator of the Rényi mutual information. A direct estimate was obtained via the estimator of Moon et al [17] with a Gaussian kernel and a bandwidth computed by Scott's rule [31].…”

Section: Comparison With the Exact Value Of The Rényi Divergencementioning

confidence: 99%

See 2 more Smart Citations

Quantifying Data Dependencies with Rényi Mutual Information and Minimum Spanning Trees

Eggels

Crommelin

2019

Entropy

View full text Add to dashboard Cite

In this study, we present a novel method for quantifying dependencies in multivariate datasets, based on estimating the Rényi mutual information by minimum spanning trees (MSTs). The extent to which random variables are dependent is an important question, e.g., for uncertainty quantification and sensitivity analysis. The latter is closely related to the question how strongly dependent the output of, e.g., a computer simulation, is on the individual random input variables. To estimate the Rényi mutual information from data, we use a method due to Hero et al. that relies on computing minimum spanning trees (MSTs) of the data and uses the length of the MST in an estimator for the entropy. To reduce the computational cost of constructing the exact MST for large datasets, we explore methods to compute approximations to the exact MST, and find the multilevel approach introduced recently by Zhong et al. (2015) to be the most accurate. Because the MST computation does not require knowledge (or estimation) of the distributions, our methodology is well-suited for situations where only data are available. Furthermore, we show that, in the case where only the ranking of several dependencies is required rather than their exact value, it is not necessary to compute the Rényi divergence, but only an estimator derived from it. The main contributions of this paper are the introduction of this quantifier of dependency, as well as the novel combination of using approximate methods for MSTs with estimating the Rényi mutual information via MSTs. We applied our proposed method to an artificial test case based on the Ishigami function, as well as to a real-world test case involving an El Nino dataset.

show abstract

Entropy estimation via uniformization

2023

Artificial Intelligence

View full text Add to dashboard Cite

Ensemble Estimation of Information Divergence †

Cited by 19 publications

References 71 publications

Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation

Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation

Quantifying Data Dependencies with Rényi Mutual Information and Minimum Spanning Trees

Entropy estimation via uniformization

Contact Info

Product

Resources

About