A first-order augmented Lagrangian method for constrained minimax optimization

Lu, Zhaosong; Mei, Sanyou

doi:10.48550/arxiv.2301.02060

Cited by 1 publication

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, Luo et al [2020] presented a class of efficient stochastic recursive GDA to solve the stochastic Non-Convex Strongly-Concave (NC-SC) minimax problems. More recently, Lu and Mei [2023] proposed a first-order augmented Lagrangian method to solve the constrained nonconvex-concave minimax problems with nonsmooth regularization. Another class of approaches is the alternating (two-timescale) GDA, which only uses a single-loop to update primal and dual variables x and y with different learning rates.…”

Section: Algorithmmentioning

confidence: 99%

Smellscapes of Nanjing Road

Huang¹

2022

Sensing China

View full text Add to dashboard Cite

Bilevel optimization is a popular two-level hierarchical optimization, which has been widely applied to many machine learning tasks such as hyperparameter learning, meta learning and continual learning. Although many bilevel optimization methods recently have been developed, the bilevel methods are not well studied when the lower-level problem is nonconvex. To fill this gap, in the paper, we study a class of nonconvex bilevel optimization problems, which both upper-level and lower-level problems are nonconvex, and the lower-level problem satisfies Polyak-Lojasiewicz (PL) condition. We propose an efficient momentum-based gradient bilevel method (MGBiO) to solve these deterministic problems. Meanwhile, we propose a class of efficient momentum-based stochastic gradient bilevel methods (MSGBiO and VR-MSGBiO) to solve these stochastic problems. Moreover, we provide a useful convergence analysis framework for our methods. Specifically, under some mild conditions, we prove that our MGBiO method has a sample (or gradient) complexity of O(ǫ −2 ) for finding an ǫ-stationary solution of the deterministic bilevel problems (i.e., ∇F (x) ≤ ǫ), which improves the existing best results by a factor of O(ǫ −1 ). Meanwhile, we prove that our MSGBiO and VR-MSGBiO methods have sample complexities of Õ(ǫ −4 ) and Õ(ǫ −3 ), respectively, in finding an ǫ-stationary solution of the stochastic bilevel problems (i.e., E ∇F (x) ≤ ǫ), which improves the existing best results by a factor of O(ǫ −3 ). This manuscript commemorates the mathematician Boris Polyak (1935 -2023).

show abstract

Section: Algorithmmentioning

confidence: 99%

Smellscapes of Nanjing Road

Huang¹

2022

Sensing China

View full text Add to dashboard Cite

show abstract

A first-order augmented Lagrangian method for constrained minimax optimization

Cited by 1 publication

References 26 publications

Smellscapes of Nanjing Road

Smellscapes of Nanjing Road

Contact Info

Product

Resources

About