Accelerated Information Gradient flow

Wang, Yifei; Li, Wuchen

doi:10.48550/arxiv.1909.02102

Cited by 6 publications

(11 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…demonstrate that underdamped LD accelerates the steepest descent steps taken by the overdamped LD, forming an analog of Nesterov acceleration for MCMC methods. Wang and Li (2019) present a framework for Nesterov's accelerated gradient method in the Wasserstein space, which consists of augmenting the energy functional with the kinetic energy of an additional momentum variable.…”

Section: Stein's Methods and Other Relevant Workmentioning

confidence: 99%

De-randomizing MCMC dynamics with the diffusion Stein operator

Shen,

Heinonen,

Kaski

2021

Preprint

View full text Add to dashboard Cite

Approximate Bayesian inference estimates descriptors of an intractable target distribution -in essence, an optimization problem within a family of distributions. For example, Langevin dynamics (LD) extracts asymptotically exact samples from a diffusion process because the time evolution of its marginal distributions constitutes a curve that minimizes the KL-divergence via steepest descent in the Wasserstein space. Parallel to LD, Stein variational gradient descent (SVGD) similarly minimizes the KL , albeit endowed with a novel Stein-Wasserstein distance, by deterministically transporting a set of particle samples, thus de-randomizes the stochastic diffusion process. We propose de-randomized kernel-based particle samplers to all diffusion-based samplers known as MCMC dynamics. Following previous work in interpreting MCMC dynamics, we equip the Stein-Wasserstein metric with a fiber-Riemannian Poisson structure, with the capacity of characterizing a fiber-gradient Hamiltonian flow that simulates MCMC dynamics. Such dynamics discretize into generalized SVGD (GSVGD), a Stein-type deterministic particle sampler, with particle updates coinciding with applying the diffusion Stein operator to a kernel function. We demonstrate empirically that GSVGD can de-randomize complicated MCMC dynamics, which combine the advantages of auxiliary momentum variables and Riemannian structure, while maintaining the high sample quality from an interacting particle system.Preprint. Under review.

show abstract

Section: Stein's Methods and Other Relevant Workmentioning

confidence: 99%

De-randomizing MCMC dynamics with the diffusion Stein operator

Shen,

Heinonen,

Kaski

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…where h is computed at the current samples w n l , e.g., as the median of their square distances (Liu and Wang, 2016) or through optimization (Wang and Li, 2019). For the step size α n l in ( 16), we use a line search technique (Chen et al, 2019b;.…”

Section: Projected Wasserstein Gradient Descentmentioning

confidence: 99%

“…The first example is a bi-modal posterior distribution with a Gaussian prior. WGD-MED and WGD-BM denote WGD with kernel bandwidth calculated by the MED method (Liu and Wang, 2016) and the BM method (Wang and Li, 2019) respectively. We compare WGD-MED, WGD-BM with SVGD.…”

Section: 1mentioning

confidence: 99%

Projected Wasserstein gradient descent for high-dimensional Bayesian inference

Wang¹,

Chen²,

Li³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a projected Wasserstein gradient descent method (pWGD) for high-dimensional Bayesian inference problems. The underlying density function of a particle system of WGD is approximated by kernel density estimation (KDE), which faces the long-standing curse of dimensionality. We overcome this challenge by exploiting the intrinsic low-rank structure in the difference between the posterior and prior distributions. The parameters are projected into a low-dimensional subspace to alleviate the approximation error of KDE in high dimensions. We formulate a projected Wasserstein gradient flow and analyze its convergence property under mild assumptions. Several numerical experiments illustrate the accuracy, convergence, and complexity scalability of pWGD with respect to parameter dimension, sample size, and processor cores.

show abstract

“…We next present the following two categories of gradient flows in Hessian density manifold. Firstly, we introduce a class of transport Newton's flows [36].…”

Section: Proof Of Claimmentioning

confidence: 99%

“…In this paper, we extend the area of TIG into the category of Hessian geometry. One direct application is to formulate optimization techniques for Bayesian sampling problems [36,37]. See related developments in information geometry [32,33].…”

Section: Introductionmentioning

confidence: 99%

Hessian metric via transport information geometry

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We propose to study the Hessian metric of given functional in the space of probability space embedded with L 2 -Wasserstein (optimal transport) metric. We name it transport Hessian metric, which contains and extends the classical L 2 -Wasserstein metric. We formulate several dynamical systems associated with transport Hessian metrics. Several connections between transport Hessian metrics and math physics equations are discovered. E.g., the transport Hessian gradient flow, including Newton's flow, formulates a mean-field kernel Stein variational gradient flow; The transport Hessian Hamiltonian flow of negative Boltzmann-Shannon entropy forms the Shallow water's equation; The transport Hessian gradient flow of Fisher information forms the heat equation. Several examples and closed-form solutions of finite-dimensional transport Hessian metrics and dynamics are presented.

show abstract

Accelerated Information Gradient flow

Cited by 6 publications

References 37 publications

De-randomizing MCMC dynamics with the diffusion Stein operator

De-randomizing MCMC dynamics with the diffusion Stein operator

Projected Wasserstein gradient descent for high-dimensional Bayesian inference

Hessian metric via transport information geometry

Contact Info

Product

Resources

About