Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks

Liang, Tengyuan; Stokes, James

doi:10.48550/arxiv.1802.06132

Cited by 34 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we only consider the statistical problem of how well GANs learn density, assuming the optimization, say (3.2), can be done to sufficient accuracy. Admittedly, computation of GANs is a considerably harder question (Mescheder, Nowozin, and Geiger, 2017;Daskalakis, Ilyas, Syrgkanis, and Zeng, 2017;Liang and Stokes, 2018;Arbel, Sutherland, Bińkowski, and Gretton, 2018;Lucic, Kurach, Michalski, Gelly, and Bousquet, 2017), which we leave as future work.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

How Well Generative Adversarial Networks Learn Distributions

Liang

2018

Preprint

Self Cite

View full text Add to dashboard Cite

We study in this paper the rate of convergence for learning distributions with the adversarial framework and Generative Adversarial Networks (GANs), which subsumes Wasserstein, Sobolev and MMD GANs as special cases. We study a wide range of parametric and nonparametric target distributions, under a collection of objective evaluation metrics. On the nonparametric end, we investigate the minimax optimal rates and fundamental difficulty of the density estimation under the adversarial framework. On the parametric end, we establish a theory for general neural network classes (including deep leaky ReLU as a special case), that characterizes the interplay on the choice of generator and discriminator. We investigate how to obtain a good statistical guarantee for GANs through the lens of regularization. We discover and isolate a new notion of regularization, called the generator/discriminator pair regularization, that sheds light on the advantage of GANs compared to classical parametric and nonparametric approaches for density estimation. We develop novel oracle inequalities as the main tools for analyzing GANs, which is of independent theoretical interest.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

How Well Generative Adversarial Networks Learn Distributions

Liang

2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Despite its popularity, this algorithm fails to converge even for simple bilinear zero-sum games [41,39,14,2,32]. This failure was fixed by adding negative momentum or by using primal-dual methods proposed by [22,21,8,13,15,33].…”

Section: Introductionmentioning

confidence: 99%

Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods

Nouiehed,

Sanjabi,

Huang

et al. 2019

Preprint

View full text Add to dashboard Cite

Recent applications that arise in machine learning have surged significant interest in solving min-max saddle point games. This problem has been extensively studied in the convex-concave regime for which a global equilibrium solution can be computed efficiently. In this paper, we study the problem in the non-convex regime and show that an ε-first order stationary point of the game can be computed when one of the player's objective can be optimized to global optimality efficiently. In particular, we first consider the case where the objective of one of the players satisfies the Polyak-Łojasiewicz (PL) condition. For such a game, we show that a simple multi-step gradient descent-ascent algorithm finds an ε-first order stationary point of the problem in O(ε −2 ) iterations. Then we show that our framework can also be applied to the case where the objective of the "max-player" is concave. In this case, we propose a multi-step gradient descent-ascent algorithm that finds an ε-first order stationary point of the game in O(ε −3.5 ) iterations, which is the best known rate in the literature. We applied our algorithm to a fair classification problem of Fashion-MNIST dataset and observed that the proposed algorithm results in smoother training and better generalization.

show abstract

“…Most of existing theories focus on the statistical properties of GANs at the global-optimum [15,16,20,87]. However, on the training side, gradient descent ascent only enjoys efficient convergence to a global optimum when the loss function is convex-concave, or efficient convergence to a critical point in general settings [37,38,48,53,71,73,75,77,78]. Due to the extreme non-linearity of the networks in both the generator and the discriminator, it is highly unlikely that the training objective of GANs can be convex-concave.…”

Section: Introductionmentioning

confidence: 99%

Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions

Allen-Zhu

2021

Preprint

View full text Add to dashboard Cite

Generative adversarial networks (GANs) are among the most successful models for learning high-complexity, real-world distributions. However, in theory, due to the highly non-convex, non-concave landscape of the minmax training objective, GAN remains one of the least understood deep learning models. In this work, we formally study how GANs can efficiently learn certain hierarchically generated distributions that are close to the distribution of images in practice. We prove that when a distribution has a structure that we refer to as forward superresolution, then simply training generative adversarial networks using gradient descent ascent (GDA) can indeed learn this distribution efficiently, both in terms of sample and time complexities. We also provide concrete empirical evidence that not only our assumption "forward super-resolution" is very natural in practice, but also the underlying learning mechanisms that we study in this paper (to allow us efficiently train GAN via GDA in theory) simulates the actual learning process of GANs in practice on real-world problems.

show abstract

Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks

Cited by 34 publications

References 0 publications

How Well Generative Adversarial Networks Learn Distributions

How Well Generative Adversarial Networks Learn Distributions

Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods

Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions

Contact Info

Product

Resources

About