Wasserstein-2 Generative Networks

Korotin, Alexander; Egiazarian, Vage; Asadulaev, Arip; Safin, Alexander; Burnaev, Evgeny

doi:10.48550/arxiv.1909.13082

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall that g satisfies Hypothesis 1. From relation (18) in Theorem 3, we get that for all θ 1 , there exists a neighborhood Ω of θ 1 such that for all θ 2 ∈ Ω…”

Section: Differentiation Of W λ Cmentioning

confidence: 99%

“…Among these extensions, the method of [19] considers generic convex costs for optimal transport and relies on low dimentional discrete transport problems on batches during the learning. In [18], the case of the 2-Wasserstein distance is tackled thanks to input convex neural networks [1] and a cycle-consistency regularization. In order to have a differentiable distance, the use of entropic regularization of optimal transport has been proposed in different ways.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Existence of Optimal Transport Gradient for Learning Generative Models

Houdard,

Leclaire,

Papadakis

et al. 2021

Preprint

View full text Add to dashboard Cite

The use of optimal transport cost for learning generative models has become popular with Wasserstein Generative Adversarial Networks (WGAN). Training of WGAN relies on a theoretical background: the calculation of the gradient of the optimal transport cost with respect to the generative model parameters. We first demonstrate that such gradient may not be defined, which can result in numerical instabilities during gradient-based optimization. We address this issue by stating a valid differentiation theorem in the case of entropic regularized transport and specify conditions under which existence is ensured. By exploiting the discrete nature of empirical data, we formulate the gradient in a semi-discrete setting and propose an algorithm for the optimization of the generative model parameters. Finally, we illustrate numerically the advantage of the proposed framework.

show abstract

“…Recall that g satisfies Hypothesis 1. From relation (18) in Theorem 3, we get that for all θ 1 , there exists a neighborhood Ω of θ 1 such that for all θ 2 ∈ Ω…”

Section: Differentiation Of W λ Cmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

On the Existence of Optimal Transport Gradient for Learning Generative Models

Houdard,

Leclaire,

Papadakis

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The leading approach -the Input Convex Neural Network (ICNN) [8] -models a convex potential which can be differentiated with respect to the inputs to produce a gradient map. Huang et al [9] combine Brenier's theorem with the ICNN gradients to design flow based density estimators, and Makkuva et al [10], Korotin et al [11] use a similar combination to solve high-dimensional barycenter and transport problems. While Huang et al [9] prove a universal approximation theorem for the ICNN, the result relies on stacking a large number of layers.…”

Section: Introductionmentioning

confidence: 99%

“…This happens because the chain rule turns the composition of layers into a product of their corresponding Jacobians. This does not cause issues for training the network on objectives involving the scalar output, like regression, but can become problematic for objectives involving the gradient of the network's output [9][10][11]. Intuitively, the product of layers of a neural network has similarities to a polynomial, and can suffer from oscillations related to the Runge phenomena -see [13].…”

Section: Introductionmentioning

confidence: 99%

Input Convex Gradient Networks

Richter-Powell¹,

Lorraine²,

Amos³

2021

Preprint

View full text Add to dashboard Cite

The gradients of convex functions are expressive models of non-trivial vector fields. For example, Brenier's theorem yields that the optimal transport map between any two measures on Euclidean space under the squared distance is realized as a convex gradient, which is a key insight used in recent generative flow models. In this paper, we study how to model convex gradients by integrating a Jacobian-vector product parameterized by a neural network, which we call the Input Convex Gradient Network (ICGN). We theoretically study ICGNs and compare them to taking the gradient of an Input-Convex Neural Network (ICNN), empirically demonstrating that a single layer ICGN can fit a toy example better than a single layer ICNN. Lastly, we explore extensions to deeper networks and connections to constructions from Riemannian geometry.

show abstract

“…To get samples from optimal coupling, the traditional methods like Linear Programming [28,34,37] or Sinkhorn [13] usually start with the discretization of the whole continuous space and compute the transport plan for discrete setting as the approximation of the continuous case. Our algorithm can directly output the sample approximation of the optimal coupling without any discretization or training process as neural network method [35,21,26]. This is also very different from other traditional methods like Monge-Ampère Equation [5] or dynamical scheme [4,24,33].…”

Section: Introductionmentioning

confidence: 99%

Approximating the Optimal Transport Plan via Particle-Evolving Method

Liu,

Sun,

Zha

2021

Preprint

View full text Add to dashboard Cite

Optimal transport (OT) provides powerful tools for comparing probability measures in various types. The Wasserstein distance which arises naturally from the idea of OT is widely used in many machine learning applications. Unfortunately, computing the Wasserstein distance between two continuous probability measures always suffers from heavy computational intractability. In this paper, we propose an innovative algorithm that iteratively evolves a particle system to match the optimal transport plan for two given continuous probability measures. The derivation of the algorithm is based on the construction of the gradient flow of an Entropy Transport Problem which could be naturally understood as a classical Wasserstein optimal transport problem with relaxed marginal constraints. The algorithm comes with theoretical analysis and empirical evidence.

show abstract

Wasserstein-2 Generative Networks

Cited by 5 publications

References 0 publications

On the Existence of Optimal Transport Gradient for Learning Generative Models

On the Existence of Optimal Transport Gradient for Learning Generative Models

Input Convex Gradient Networks

Approximating the Optimal Transport Plan via Particle-Evolving Method

Contact Info

Product

Resources

About