Training GANs with centripetal acceleration

Peng, Wei Jei; Dai, Yu‐Hong; Zhang, Hui; Cheng, Lizhi

doi:10.1080/10556788.2020.1754414

Cited by 24 publications

(33 citation statements)

References 11 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…where γ t is sometimes called the optimism rate. Similarly to our conclusions, it has been empirically observed that taking large optimism rate often yields better convergence in stochastic problems [29].…”

Section: Beyond Extragradientsupporting

confidence: 91%

Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling

Hsieh,

Iutzeler,

Malick

et al. 2020

Preprint

View full text Add to dashboard Cite

Owing to their stability and convergence speed, extragradient methods have become a staple for solving large-scale saddle-point problems in machine learning. The basic premise of these algorithms is the use of an extrapolation step before performing an update; thanks to this exploration step, extragradient methods overcome many of the non-convergence issues that plague gradient descent/ascent schemes. On the other hand, as we show in this paper, running vanilla extragradient with stochastic gradients may jeopardize its convergence, even in simple bilinear models. To overcome this failure, we investigate a double stepsize extragradient algorithm where the exploration step evolves at a more aggressive time-scale compared to the update step. We show that this modification allows the method to converge even with stochastic gradients, and we derive sharp convergence rates under an error bound condition.

show abstract

Section: Beyond Extragradientsupporting

confidence: 91%

Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling

Hsieh,

Iutzeler,

Malick

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The first variant of Extra-Gradient with a single oracle call per iteration dates back to Popov [38]. This algorithm was subsequently studied by, among others, Chiang et al [10], Rakhlin and Sridharan [39,40] and Gidel et al [19]; see also [14,26] for a "reflected" variant, [15,31,32,37] for an "optimistic" one, and Section 3 for a discussion of the differences between these variants. In the context of deterministic, strongly monotone variational inequalities with Lipschitz continuous operators, the last iterate of the method was shown to exhibit a geometric convergence rate [19,26,32,43]; similar geometric convergence results also extend to bilinear saddlepoint problems [19,37,43], even though the operator involved is not strongly monotone.…”

Section: Related Workmentioning

confidence: 99%

On the convergence of single-call stochastic extra-gradient methods

Hsieh,

Iutzeler,

Malick

et al. 2019

Preprint

View full text Add to dashboard Cite

Variational inequalities have recently attracted considerable interest in machine learning as a flexible paradigm for models that go beyond ordinary loss function minimization (such as generative adversarial networks and related deep learning systems). In this setting, the optimal O(1/t) convergence rate for solving smooth monotone variational inequalities is achieved by the Extra-Gradient (EG) algorithm and its variants. Aiming to alleviate the cost of an extra gradient step per iteration (which can become quite substantial in deep learning applications), several algorithms have been proposed as surrogates to Extra-Gradient with a single oracle call per iteration. In this paper, we develop a synthetic view of such algorithms, and we complement the existing literature by showing that they retain a O(1/t) ergodic convergence rate in smooth, deterministic problems. Subsequently, beyond the monotone deterministic case, we also show that the last iterate of single-call, stochastic extra-gradient methods still enjoys a O(1/t) local convergence rate to solutions of non-monotone variational inequalities that satisfy a second-order sufficient condition.

show abstract

“…At last, we would like to point out that the BEP method does not cover the simultaneous centripetal acceleration and alternating centripetal acceleration methods, proposed in our recent work [21] for training GANs, one of which also includes the OGDA as a special case.…”

Section: Bregman Extrapolation Methodsmentioning

confidence: 99%

Extragradient and Extrapolation Methods with Generalized Bregman Distances for Saddle Point Problems

Zhang¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this work, we introduce two algorithmic frameworks, named Bregman extragradient method and Bregman extrapolation method, for solving saddle point problems. The proposed frameworks not only include the well-known extragradient and optimistic gradient methods as special cases, but also generate new variants such as sparse extragradient and extrapolation methods. With the help of the recent concept of relative Lipschitzness and some Bregman distance related tools, we are able to show certain upper bounds in terms of Bregman distances for "regret" measures. Further, we use those bounds to deduce the convergence rate of O(1/k) for the Bregman extragradient and Bregman extrapolation methods applied to solving smooth convex-concave saddle point problems. Our theory recovers the main discovery made in [Mokhtari et al. (2020), SIAM J. Optim., 20, pp. 3230-3251] for more general algorithmic frameworks with weaker assumptions via a conceptually different approach.

show abstract

Training GANs with centripetal acceleration

Cited by 24 publications

References 11 publications

Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling

Explore Aggressively, Update Conservatively: Stochastic Extragradient Methods with Variable Stepsize Scaling

On the convergence of single-call stochastic extra-gradient methods

Extragradient and Extrapolation Methods with Generalized Bregman Distances for Saddle Point Problems

Contact Info

Product

Resources

About