Information geometry of divergence functions

In this paper, we extend and overview wide families of Alpha-, Beta-and Gamma-divergences and discuss their fundamental properties. In literature usually only one single asymmetric (Alpha, Beta or Gamma) divergence is considered. We show in this paper that there exist families of such divergences with the same consistent properties. Moreover, we establish links and correspondences among these divergences by applying suitable nonlinear transformations. For example, we can generate the Beta-divergences directly from Alpha-divergences and vice versa. Furthermore, we show that a new wide class of Gamma-divergences can be generated not only from the family of Beta-divergences but also from a family of Alpha-divergences. The paper bridges these divergences and shows also their links to Tsallis and Rényi entropies. Most of these divergences have a natural information theoretic interpretation.

show abstract

“…An insight using information geometry may further elucidates the fundamental structures of such divergences and geometry [1,3,7].…”

Section: Conclusion and Discussionmentioning

confidence: 99%

“…The divergences are closely related to the invariant geometrical properties of the manifold of probability distributions [5][6][7].…”

Section: D(p || Z) ≤ D(p || Q) + D(q || Z) (Subaddivity/triangle Ineqmentioning

confidence: 99%

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Cichocki

Амари

2010

Entropy

381

313

View full text Add to dashboard Cite

show abstract

“…The proof of equivalence uses concepts in convex analysis combined with connections between Bregman divergences and Riemannian manifolds developed in [2]. Using the equivalence of the two algorithms, we can exploit the desireable properties of both algorithms.…”

Section: Our Contributionmentioning

confidence: 99%

“…In particular, let G : Θ → R denote a strictly convex twice-differentiable function, the divergence introduced by [7] B G : Θ × Θ → R + is: Bregman divergences are widely used in statistical inference, optimization, machine learning, and information geometry (see e.g. [2,5]). Letting Ψ(·, ·) = B G (·, ·), the mirror descent step defined is:…”

Section: Mirror Descent With Bregman Divergencesmentioning

confidence: 99%

The Information Geometry of Mirror Descent

Raskutti

Mukherjee

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We prove the equivalence of two online learning algorithms, mirror descent and natural gradient descent. Both mirror descent and natural gradient descent are generalizations of online gradient descent when the parameter of interest lies on a non-Euclidean manifold. Natural gradient descent selects the steepest descent along a Riemannian manifold by multiplying the standard gradient by the inverse of the metric tensor. Mirror descent induces non-Euclidean structure by solving iterative optimization problems using different proximity functions. In this paper, we prove that mirror descent induced by a Bregman divergence proximity functions is equivalent to the natural gradient descent algorithm on the dual Riemannian manifold. We use use techniques from convex analysis and a connections between Riemannian manifolds, Bregman divergences and convexity to prove this result. This equivalence between natural gradient descent and mirror descent, implies that (1) mirror descent is the steepest descent direction along the Riemannian manifold corresponding to the choice of Bregman divergence; (2) mirror descent with log-likelihood loss applied to parameter estimation in exponential families asymptotically achieves the classical Cramér-Rao lower bound, and (3) natural gradient descent for manifolds corresponding to exponential families can be implemented as a first-order method through mirror descent.

show abstract

“…But J. Burbea and C. R. Rao have proved in [18,Theorem 2] that the Bergman metric and the Fisher information metric do coincide for some probability density functions of particular forms. A similar potential function was used by S. Amari in [2] to derive the Riemannian metric of multi-variate Gaussian distributions by means of divergence functions. We refer to [29] for more account on the geometry of Hessian structures.…”

Section: Riemannian Geometry Of Toeplitz Covariance Matricesmentioning

confidence: 99%

Medians and Means in Riemannian Geometry: Existence, Uniqueness and Computation

Arnaudon

Barbaresco

Yang

2012

Matrix Information Geometry

View full text Add to dashboard Cite

This paper is a short summary of our recent work on the medians and means of probability measures in Riemannian manifolds. Firstly, the existence and uniqueness results of local medians are given. In order to compute medians in practical cases, we propose a subgradient algorithm and prove its convergence. After that, Fréchet medians are considered. We prove their statistical consistency and give some quantitative estimations of their robustness with the aid of upper curvature bounds. We also show that, in compact Riemannian manifolds, the Fréchet medians of generic data points are always unique. Stochastic and deterministic algorithms are proposed for computing Riemannian p-means. The rate of convergence and error estimates of these algorithms are also obtained. Finally, we apply the medians and the Riemannian geometry of Toeplitz covariance matrices to radar target detection.

show abstract

Information geometry of divergence functions

Cited by 107 publications

References 25 publications

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

Families of Alpha- Beta- and Gamma- Divergences: Flexible and Robust Measures of Similarities

The Information Geometry of Mirror Descent

Medians and Means in Riemannian Geometry: Existence, Uniqueness and Computation

Contact Info

Product

Resources

About