Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)

Abstract

Wasserstein GANs are based on the idea of minimising the Wasserstein distance between a real and a generated distribution. We provide an indepth mathematical analysis of differences between the theoretical setup and the reality of training Wasserstein GANs. In this work, we gather both theoretical and empirical evidence that the WGAN loss is not a meaningful approximation of the Wasserstein distance. Moreover, we argue that the Wasserstein distance is not even a desirable loss function for deep generative models, and conclude that the success of Wasserstein GANs can in truth be attributed to a failure to approximate the Wasserstein distance.

Wasserstein GANs with Gradient Penalty Compute Congested Transport

“…This is in contrast to the standard Wasserstein 1 framework, where the gradients of a solution u to (1) determine only the direction of optimal mass transport, and not the speed of that transport or the amount of mass transported. This insight provides more evidence to support the hypothesis in Stanczuk et al (2021) that WGAN-GP might perform well because they fail to compute W 1 (µ, ν).…”

“…This is in contrast to the gradients of Kantorovich potentials for the Wasserstein 1 distance, which only determine the normalized direction of flow. This may explain, in support of Stanczuk et al (2021), the success of WGAN-GP, since the training of the generator is based on these gradients.…”

“…More generally, several papers (e.g. Mallasto et al (2019), Pinetz et al (2019), Stanczuk et al (2021)) have provided empirical evidence that WGAN-GP do not compute W 1 (µ, ν), but do not offer a precise notion of what they do compute. In particular, Mallasto et al (2019) shows that for simple discrete problems WGAN-GP tend to over estimate the Wasserstein 1 distance; statement 2 of our Theorem A provides some theoretical backing for this observation.…”

“…While WGAN-GP have enjoyed spectacular success, the question of whether they are actually computing the Wasserstein 1 distance has only been studied more recently in, for example, Mallasto et al (2019), Pinetz et al (2019), and Stanczuk et al (2021). In particular we were intrigued by Stanczuk et al (2021), which offers empirical evidence that WGAN-GP do not compute W 1 (µ, ν), and, due to some issues with the Wasserstein 1 distance, argue that this might be the reason for their success. In this paper we analyse this further by establishing for the first time the quantity that WGAN-GP with one-sided penalty do compute: it is a congested transport distance.…”

“…While WGAN-GP were initially developed to calculate the Wasserstein 1 distance between generated and real data, recent works (e.g. Stanczuk et al (2021)) have provided empirical evidence that this does not occur, and have argued that WGAN-GP perform well not in spite of this issue, but because of it. In this paper we show for the first time that WGAN-GP compute the minimum of a different optimal transport problem, the so-called congested transport (Carlier et al (2008)).…”