Statistical Inference Based on Divergence Measures

Pardo, Leonardo

doi:10.1201/9781420034813

Cited by 311 publications

(279 citation statements)

References 0 publications

Supporting

Mentioning

273

Contrasting

Unclassified

Order By: Relevance

“…(As a warning, note that most sources show erroneous expressions for the Chernoff and/or Rényi α-divergence between two multivariate Gaussians, including [27,29,[38][39][40], and even a late draft of this manuscript.) For the upper boundĤ KL , the KL divergence between two multivariate Gaussians…”

Section: Gaussian Mixturesmentioning

confidence: 99%

Estimating Mixture Entropy with Pairwise Distances

Kolchinsky

Tracey

2017

Entropy

109

View full text Add to dashboard Cite

Mixture distributions arise in many parametric and non-parametric settings-for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff α-divergence gives a lower bound when chosen as the distance function, with the Bhattacharyaa distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback-Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.

show abstract

Section: Gaussian Mixturesmentioning

confidence: 99%

Estimating Mixture Entropy with Pairwise Distances

Kolchinsky

Tracey

2017

Entropy

109

View full text Add to dashboard Cite

show abstract

“…Estimation of divergence and its applications have been many studies using different approaches. For example Pardo [20] presented methods and applications in the case of discrete distributions. By exploring a nonparametric method for estimating the divergence in the continuous case, Poczos and Schneider [23] proposed a -nearest-neighbor estimator and proved the weak consistency of the estimator Rényi-and Tsallisdivergences.…”

Section: Introductionmentioning

confidence: 99%

Kernel-Type Estimators of Divergence Measures and Its Strong Uniform Consistency

Dhaker

Ngom

Dème³

et al. 2016

AJTAS

View full text Add to dashboard Cite

Nonparametric density estimation, based on kernel-type estimators, is a very popular method in statistical research, especially when we want to model the probabilistic or stochastic structure of a data set. In this paper, we investigate the asymptotic confidence bands for the distribution with kernel-estimators for some types of divergence measures (Rényi-α and Tsallis-α divergence). Our aim is to use the method based on empirical process techniques, in order to derive some asymptotic results. Under different assumptions, we establish a variety of fundamental and theoretical properties, such as the strong consistency of an uniform-in-bandwidth of the divergence estimators. We further apply the previous results in simulated examples, including the kernel-type estimator for Hellinger, Bhattacharyya and Kullback-Leibler divergence, to illustrate this approach, and we show that that the method performs competitively.

show abstract

“…For these models, among the criteria shown above, the AIC, TIC, BIC and CVC can be used, where the likelihood based on multinomial or categorical distributions are used for the criteria except the CVC. The  -divergence statistic (see e.g., Cressie & Pardo, 2002a;Pardo, 2006) is a generalization of the log-likelihood ratio statistic for evaluating the goodness-of-fit of a model. While the original definition of the AIC is based on the likelihood rather than the likelihood ratio, the latter can also be used for model selection in essentially the same way, since an added term in the log-likelihood ratio common to candidate models is irrelevant to model selection.…”

Section: Introductionmentioning

confidence: 99%

A family of the information criteria using the phi-divergence for categorical data

Ogasawara

2018

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

The risk of the phi-divergence of a statistical model for categorical data is defined using two independent sets of data. The asymptotic bias of the phi-divergence based on current data as an estimator of the risk is shown to be equal to the negative penalty term of the Akaike information criterion (AIC). Though the higher-order asymptotic bias is derived, the higher-order bias depends on the form of the phi-divergence and the estimation method of parameters using a possible different form of the phi-divergence. An approximation to the higher-order bias is obtained based on the simple result of the saturated model. The information criteria using this approximation yield improved results in simulations for model selection. Some cases of the power divergences forming a subfamily of the phi-divergence show advantages over the AIC in simulations.

show abstract

Statistical Inference Based on Divergence Measures

Cited by 311 publications

References 0 publications

Estimating Mixture Entropy with Pairwise Distances

Estimating Mixture Entropy with Pairwise Distances

Kernel-Type Estimators of Divergence Measures and Its Strong Uniform Consistency

A family of the information criteria using the phi-divergence for categorical data

Contact Info

Product

Resources

About