Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler Divergence

Noh, Yung‐Kyun; Sugiyama, Masashi; Liu, Song; Plessis, Marthinus Christoffel du; Park, Frank Chongwoo; Lee, Daniel D.

doi:10.1162/neco_a_01092

Cited by 20 publications

(23 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, Noh et al (2014) showed that the bias of the NNDE-based KL-divergence approximator at x is approximately proportional to…”

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

“…One way to reduce the bias is to learn appropriate distance metric in the KL-divergence approximator, equation 4.1, while the KL-divergence itself is metric invariant. In Noh et al (2014), the best local Mahalanobis metric (x − x ) A(x − x ) at x that minimizes the estimated bias was given as the solution of the following optimization problem:…”

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

“…The matrices A and B share the same eigenvectors, and U + ∈ R d×d + and U − ∈ R d×d − are collections of the eigenvectors corresponding to the eigenvalues in + and − , respectively. To compute the bias matrix B, Noh et al (2014) assumed that p 1 and p 2 are both nearly gaussian, and estimated densities p 1 and p 2 as well as their Hessian matrices ∇ 2 p 1 and ∇ 2 p 2 using the gaussian parametric model with maximum likelihood estimation. The accuracy of NNDE-based KLdivergence approximation was significantly improved when p 1 and p 2 are nearly gaussian.…”

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

“…r The bias of nearest-neighbor Kullback-Leibler (KL) divergence approximation depends on the second-order density derivatives (Noh et al, 2014).…”

mentioning

confidence: 99%

“…Unlike previous work (Noh et al, 2014;Duong & Hazelton, 2005;Chacón & Duong, 2010), in both applications, we estimate the second-order derivatives directly by ISED without making use of any parametric models or any assumptions on the data density. The experimental results confirm the usefulness of ISED on various tasks, such as change detection, feature selection, and density estimation.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Direct Density Derivative Estimation

et al. 2016

Self Cite

View full text Add to dashboard Cite

Estimating the derivatives of probability density functions is an essential step in statistical data analysis. A naive approach to estimate the derivatives is to first perform density estimation and then compute its derivatives. However, this approach can be unreliable because a good density estimator does not necessarily mean a good density derivative estimator. To cope with this problem, in this letter, we propose a novel method that directly estimates density derivatives without going through density estimation. The proposed method provides computationally efficient estimation for the derivatives of any order on multidimensional data with a hyperparameter tuning method and achieves the optimal parametric convergence rate. We further discuss an extension of the proposed method by applying regularized multitask learning and a general framework for density derivative estimation based on Bregman divergences. Applications of the proposed method to nonparametric Kullback-Leibler divergence approximation and bandwidth matrix selection in kernel density estimation are also explored.

show abstract

“…Indeed, Noh et al (2014) showed that the bias of the NNDE-based KL-divergence approximator at x is approximately proportional to…”

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

Section: Metric Learning With Parametric Bias Estimation For Nnde-basmentioning

confidence: 99%

“…r The bias of nearest-neighbor Kullback-Leibler (KL) divergence approximation depends on the second-order density derivatives (Noh et al, 2014).…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Direct Density Derivative Estimation

et al. 2016

Self Cite

View full text Add to dashboard Cite

show abstract

A Distance for HMMs Based on Aggregated Wasserstein Metric and State Registration

Chen

2016

Computer Vision – ECCV 2016

View full text Add to dashboard Cite

We propose a framework, named Aggregated Wasserstein, for computing a dissimilarity measure or distance between two Hidden Markov Models with state conditional distributions being Gaussian. For such HMMs, the marginal distribution at any time spot follows a Gaussian mixture distribution, a fact exploited to softly match, aka register, the states in two HMMs. We refer to such HMMs as Gaussian mixture model-HMM (GMM-HMM). The registration of states is inspired by the intrinsic relationship of optimal transport and the Wasserstein metric between distributions. Specifically, the components of the marginal GMMs are matched by solving an optimal transport problem where the cost between components is the Wasserstein metric for Gaussian distributions. The solution of the optimization problem is a fast approximation to the Wasserstein metric between two GMMs. The new Aggregated Wasserstein distance is a semi-metric and can be computed without generating Monte Carlo samples. It is invariant to relabeling or permutation of the states. This distance quantifies the dissimilarity of GMM-HMMs by measuring both the di↵erence between the two marginal GMMs and the difference between the two transition matrices. Our new distance is tested on the tasks of retrieval and classification of time series. Experiments on both synthetic data and real data have demonstrated its advantages in terms of accuracy as well as e ciency in comparison with existing distances based on the Kullback-Leibler divergence.

show abstract