Max-Sliced Wasserstein Distance and Its Use for GANs

Abstract

Generative adversarial nets (GANs) and variational auto-encoders have significantly improved our distribution modeling capabilities, showing promise for dataset augmentation, image-to-image translation and feature learning. However, to model high-dimensional distributions, sequential training and stacked architectures are common, increasing the number of tunable hyper-parameters as well as the training time. Nonetheless, the sample complexity of the distance metrics remains one of the factors affecting GAN training. We first show that the recently proposed sliced Wasserstein distance has compelling sample complexity properties when compared to the Wasserstein distance. To further improve the sliced Wasserstein distance we then analyze its 'projection complexity' and develop the max-sliced Wasserstein distance which enjoys compelling sample complexity while reducing projection complexity, albeit necessitating a max estimation. We finally illustrate that the proposed distance trains GANs on high-dimensional images up to a resolution of 256x256 easily.

Strong equivalence between metrics of Wasserstein type

“…To tackle the issue of complexity, a sliced version of the Wasserstein distance was studied and employed, which only requires estimating distances of projected uni-dimensional distributions and is, therefore, more efficient, see e.g. [3], [6], [13].…”

“…[15], [3], [18]. Very recently, in order to reduce the projection complexity of the sliced Wasserstein, [6] introduced the so-called max-Wasserstein metrics, which we will denote by W p , as a fix. The same paper also points out that both of these projected versions of the Wasserstein distance enjoy the so-called generalizability over the Wasserstein metric.…”

“…

The sliced Wasserstein metric Wp and more recently max-sliced Wasserstein metric Wp have attracted abundant attention in data sciences and machine learning due to their advantages to tackle the curse of dimensionality, see e.g. [15], [6]. A question of particular importance is the strong equivalence between these projected Wasserstein metrics and the (classical) Wasserstein metric Wp.

…”

Hilbert Sinkhorn Divergence for Optimal Transport

“…An alternative principle for approximating the Wasserstein distance comes from Radon transform: to project high-dimensional distribution to one-dimensional distributions. One representative example is the sliced Wasserstein distance [8,23,14,24], which is defined as the average Wasserstein distance obtained between random one dimensional projections. In other words, the sliced Wasserstein distance is calculated via linear slicing of the probability distribution.…”

“…Its important extensions, such as [34,31], are proposed recently to search for the k-dimensional subspace that would maximize the Wasserstein distance between two measures after projection. The sample complexity of such estimators is investigated [15,14] between two measures and their empirical counterparts.…”