Ensemble estimation of multivariate f-divergence

Abstract

f -divergence estimation is an important problem in the fields of information theory, machine learning, and statistics. While several divergence estimators exist, relatively few of their convergence rates are known. We derive the MSE convergence rate for a density plug-in estimator of f -divergence. Then by applying the theory of optimally weighted ensemble estimation, we derive a divergence estimator with a convergence rate of O 1 T that is simple to implement and performs well in high dimensions. We validate our theoretical results with experiments.I. INTRODUCTION f -divergence is a measure of the difference between distributions and is important to the fields of information theory, machine learning, and statistics [1]. Many different kinds of f -divergences have been defined including the Kullback-Leibler (KL) [2] and . A special case of the KL divergence is mutual information which gives the capacities in data compression and channel coding [4]. Mutual information estimation has also been used in applications such as feature selection [5], fMRI data processing [6], and clustering [7]. Entropy is also a special case of divergence where one of the distributions is the uniform distribution. Entropy estimation is useful for intrinsic dimension estimation [8], texture classification and image registration [9], and many other applications. Additionally, divergence estimation is useful for empirically estimating the decay rates of error probabilities of hypothesis testing [4] and extending machine learning algorithms to distributional features [10], [11]. For other applications of divergence estimation, see [12].We consider the problem of estimating the f -divergence when only two finite populations of independent and identically distributed (i.i.d.) samples are available from some unknown, nonparametric, smooth, d-dimensional distributions. While several estimators of divergence have been previously defined, the convergence rates are known for only a few of them. Our first contribution is to derive convergence rates for kernel density plug-in f -divergence estimators with an adaptive k-nearest neighbor (k-nn) kernel. Our second contribution is to extend the theory of optimally weighted ensemble entropy estimation developed in [13] to obtain a divergence estimator with a convergence rate of O

Divergence Measures Estimation and its Asymptotic Normality Theory in the Discrete Case

“…There has been recent interest in deriving convergence rates for divergence estimators Moon andHero (2014)-Krishnamurthy et al (2014). The rates are typically derived in terms of smoothness s of the densities :…”

Image patch analysis of sunspots and active regions

“…we use the nonparametric estimator derived in Moon & Hero III (2014a, 2014b that is based on the k-nearest neighbor density estimators for the densities f i and f j . This estimator is simple to implement and achieves the parametric convergence rate when the densities are sufficiently smooth.…”

“…We then estimate the Hellinger distance H(f a , f b ) using the divergence estimator in Moon & Hero III (2014a).…”

Scalable Hash-Based Estimation of Divergence Measures

“…Information theoretic measures such as Shannon entropy, mutual information, and the Kullback-Leibler (KL) divergence have a broad range of applications in information theory, statistics and machine learning [1][2][3]. When we have two or more data sets and we are interested in finding the correlation or dissimilarity between them, Shannon mutual information or KL-divergence is often used.…”