Generative Modeling Using the Sliced Wasserstein Distance

Abstract

Generative Adversarial Nets (GANs) are very successful at modeling distributions from given samples, even in the high-dimensional case. However, their formulation is also known to be hard to optimize and often not stable. While this is particularly true for early GAN formulations, there has been significant empirically motivated and theoretically founded progress to improve stability, for instance, by using the Wasserstein distance rather than the Jenson-Shannon divergence. Here, we consider an alternative formulation for generative modeling based on random projections which, in its simplest form, results in a single objective rather than a saddle-point formulation. By augmenting this approach with a discriminator we improve its accuracy. We found our approach to be significantly more stable compared to even the improved Wasserstein GAN. Further, unlike the traditional GAN loss, the loss formulated in our method is a good measure of the actual distance between the distributions and, for the first time for GAN training, we are able to show estimates for the same.

Max-Sliced Wasserstein Distance and Its Use for GANs

“…In practice, Deshpande et al [8] approximate the sliced Wasserstein-2 distance between the distributions by using samples D ∼ P d , F ∼ P g , and a finite number of random Gaussian directions, replacing the integration over Ω with a summation over a randomly chosen set of unit vec-torsΩ ∝ N (0, I), where '∝' is used to indicate normalization to unit length. With P g (and hence, F) being implicitly parametrized by θ g , [8] uses the following program for generative modeling:…”

Realistic River Image Synthesis using Deep Generative Adversarial Networks

“…A. Laplacian SWD SWD, or Sliced Wasserstein Distance, is a metric that measures the overall deviation between the original training dataset and the generator-synthesized dataset [46]. The standard Wasserstein Distance is difficult to compute on such high-dimensional input, especially in images due to the 3color channel RGB values attached to each tensor.…”

“…The standard Wasserstein Distance is difficult to compute on such high-dimensional input, especially in images due to the 3color channel RGB values attached to each tensor. Each image is turned into a Laplacian pyramid [46], a multi-scaled data structure with layers representing each resolution that the image was generated at. To cross resolutions, there are upsampling, downsampling, and blurring functions applied to the original large image, which are then all compiled into one file.…”

“…The Laplacian SWD metric converts high-dimensional inputs into one-dimensional distributions, which are randomly sampled across the image and then added up as a loss function. We computed the Laplacian SWD [46], [47] score for key resolutions across the PGGANs training (specifically resolutions 32 → 256). As seen in Table II, these normalized values tended to significantly decrease over time as the resolutions grew.…”

Sliced Wasserstein Kernels for Probability Distributions

“…One of the thought-provoking approaches is via slicing high-dimensional distributions over their onedimensional marginals and comparing their marginal distributions Kolouri et al (2019b); Nadjahi et al (2019b). The idea of slicing distributions is related to the Radon transform and has been successfully used in, for instance, sliced-Wasserstein distances in various applications Rabin et al (2011); Kolouri et al (2016); Carriere et al (2017); Deshpande et al (2018); Kolouri et al (2018); Nadjahi et al (2019a). More recently, Kolouri et al (2019a) extended the idea of linear slices of distributions, used in sliced-Wasserstein distances, to non-linear slicing of high-dimensional distributions, which is rooted in the generalized Radon transform.…”