Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization

Zhou, Yi; Wang, Zhe; Ji, Kaiyi; Liang, Y. T.; Tarokh, Vahid

doi:10.24963/ijcai.2020/201

Cited by 7 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To provide a neat version of thesis with closely correlated topics, this thesis does not include all of the author's works. We briefly talk about some representatives of the author's other research works [56,64,59,123,61,60,125,101,116,47,130,129,133] as follows.…”

Section: Other Phd Researchmentioning

confidence: 99%

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Bilevel optimization has become a powerful framework in a variety of machine learning applications including signal processing, meta-learning, hyperparameter optimization, reinforcement learning and network architecture search. There are generally two classes of bilevel optimization formulations for modern machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first problem class, two popular types of gradient-based algorithms have been proposed to estimate the gradient of the outer-level objective (hypergradient) via approximate implicit differentiation (AID) and iterative differentiation (ITD). Algorithms for the second problem class include the popular model-agnostic meta-learning (MAML) and almost no inner loop (ANIL). Although bilevel optimization algorithms have been widely used, their convergence rate and fundamental limitations have not been well explored.

show abstract

Section: Other Phd Researchmentioning

confidence: 99%

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, it also generalizes other global geometries such as strong convexity and PŁ geometry. In the existing literature, the KŁ geometry has been exploited extensively to analyze the convergence rate of various gradient-based algorithms in nonconvex optimization, e.g., gradient descent Attouch and Bolte (2009); Li et al (2017) and its accelerated version Zhou et al (2020) as well as the distributed version Zhou et al (2016a). Hence, we are highly motivated to study the convergence rate of variable convergence of GDA in nonconvex minimax optimization under the KŁ geometry.…”

Section: Introductionmentioning

confidence: 99%

Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry

Chen¹,

Zhou²,

Xu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The gradient descent-ascent (GDA) algorithm has been widely applied to solve minimax optimization problems. In order to achieve convergent policy parameters for minimax optimization, it is important that GDA generates convergent variable sequences rather than convergent sequences of function values or gradient norms. However, the variable convergence of GDA has been proved only under convexity geometries, and there lacks understanding for general nonconvex minimax optimization. This paper fills such a gap by studying the convergence of a more general proximal-GDA for regularized nonconvex-strongly-concave minimax optimization. Specifically, we show that proximal-GDA admits a novel Lyapunov function, which monotonically decreases in the minimax optimization process and drives the variable sequence to a critical point. By leveraging this Lyapunov function and the KŁ geometry that parameterizes the local geometries of general nonconvex functions, we formally establish the variable convergence of proximal-GDA to a critical point x * , i.e., x t → x * , y t → y * (x * ). Furthermore, over the full spectrum of the KŁ-parameterized geometry, we show that proximal-GDA achieves different types of convergence rates ranging from sublinear convergence up to finite-step convergence, depending on the geometry associated with the KŁ parameter. This is the first theoretical result on the variable convergence for nonconvex minimax optimization.

show abstract

Event-Triggered Acceleration Algorithms for Distributed Stochastic Optimization

Lü

Liao

et al. 2022

Wireless Networks

View full text Add to dashboard Cite

Proximal Gradient Algorithm with Momentum and Flexible Parameter Restart for Nonconvex Optimization

Cited by 7 publications

References 7 publications

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Proximal Gradient Descent-Ascent: Variable Convergence under KŁ Geometry

Event-Triggered Acceleration Algorithms for Distributed Stochastic Optimization

Contact Info

Product

Resources

About