An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems

Hien, Le Thi Khanh; Zhao, Renbo; Haskell, William B.

doi:10.48550/arxiv.1711.03669

Cited by 10 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Can this work be extended to saddle point problem? In particular, [19] discussed an inexact primal-dual method for nonbilinear saddle point problems with bounded dom(g). Can we get rid of the boundedness assumption for saddle point problem?…”

Section: Conclusion and Future Researchmentioning

confidence: 99%

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

Li¹,

Qin²

2019

Preprint

View full text Add to dashboard Cite

We propose an inexact proximal augmented Lagrangian framework with explicit inner problem termination rule for composite convex optimization problems. We consider arbitrary linearly convergent inner solver including in particular stochastic algorithms, making the resulting framework more scalable facing the everincreasing problem dimension. Each subproblem is solved inexactly with an explicit and self-adaptive stopping criterion, without requiring to set an a priori target accuracy. When the primal and dual domain are bounded, our method achieves O(1/ √ ǫ) and O(1/ǫ) complexity bound in terms of number of inner solver iterations, respectively for the strongly convex and non-strongly convex case. Without the boundedness assumption, only logarithm terms need to be added and the above two complexity bounds increase respectively to Õ(1/ √ ǫ) and Õ(1/ǫ), which hold both for obtaining ǫ-optimal and ǫ-KKT solution. Within the general framework that we propose, we also obtain Õ(1/ǫ) and Õ(1/ǫ 2 ) complexity bounds under relative smoothness assumption on the differentiable component of the objective function. We show through theoretical analysis as well as numerical experiments the computational speedup possibly achieved by the use of randomized inner solvers for large-scale problems.

show abstract

Section: Conclusion and Future Researchmentioning

confidence: 99%

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

Li¹,

Qin²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The above inequality together with 2ρ x S − x S−1 ≤ ε 2 gives dist(0, ∂Φ(x S )) ≤ ε, which implies that x S is an ε-stationary point to (8).…”

Section: Inexact Proximal Point Methods (Ippm) For Nonconvex Composit...mentioning

confidence: 97%

“…FOMs have also been proposed for minimax problems. For example, [6,8] study FOMs for convex-concave minimax problems, and [15,17,18] analyzes FOMs for nonconvex-concave minimax problems. While a nonlinearconstrained optimization problem can be formulated as a minimax problem, its KKT conditions are stronger than the stationarity conditions of a nonconvex-concave minimax problem, because the latter with a compact dual domain cannot guarantee primal feasibility.…”

Section: Related Workmentioning

confidence: 99%

Rate-improved Inexact Augmented Lagrangian Method for Constrained Nonconvex Optimization

Li,

Chen,

Liu

et al. 2020

Preprint

View full text Add to dashboard Cite

First-order methods have been studied for nonlinear constrained optimization within the framework of the augmented Lagrangian method (ALM) or penalty method. We propose an improved inexact ALM (iALM) and conduct a unified analysis for nonconvex problems with either convex or nonconvex constraints. Under certain regularity conditions (that are also assumed by existing works), we show an Õ(ε − 5 2 ) complexity result for a problem with a nonconvex objective and convex constraints and an Õ(ε −3 ) complexity result for a problem with a nonconvex objective and nonconvex constraints, where the complexity is measured by the number of first-order oracles to yield an ε-KKT solution. Both results are the best known. The same-order complexity results have been achieved by penalty methods. However, two different analysis techniques are used to obtain the results, and more importantly, the penalty methods generally perform significantly worse than iALM in practice. Our improved iALM and analysis close the gap between theory and practice. Numerical experiments are provided to demonstrate the effectiveness of our proposed method.

show abstract

“…Algorithms for finding equilibria in games Since the pioneering work of von Neumann (1928), equilibria in games have received great attention. Most past results focus on convex-concave settings (Chen et al, 2014;Hien et al, 2017). Notably, Cherukuri et al (2017) studied convergence of the GDA algorithm under strictly convex-concave assumptions.…”

Section: Stochastic Estimates Of the Functionmentioning

confidence: 99%

Direct-Search for a Class of Stochastic Min-Max Problems

Anagnostidis¹,

Lucchi²,

Diouane³

2021

Preprint

View full text Add to dashboard Cite

Recent applications in machine learning have renewed the interest of the community in min-max optimization problems. While gradient-based optimization methods are widely used to solve such problems, there are however many scenarios where these techniques are not well-suited, or even not applicable when the gradient is not accessible. We investigate the use of direct-search methods that belong to a class of derivative-free techniques that only access the objective function through an oracle. In this work, we design a novel algorithm in the context of min-max saddle point games where one sequentially updates the min and the max player. We prove convergence of this algorithm under mild assumptions, where the objective of the max-player satisfies the Polyak-Lojasiewicz (PL) condition, while the min-player is characterized by a nonconvex objective. Our method only assumes dynamically adjusted accurate estimates of the oracle with a fixed probability. To the best of our knowledge, our analysis is the first one to address the convergence of a direct-search method for min-max objectives in a stochastic setting.

show abstract

An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems

Cited by 10 publications

References 0 publications

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

Rate-improved Inexact Augmented Lagrangian Method for Constrained Nonconvex Optimization

Direct-Search for a Class of Stochastic Min-Max Problems

Contact Info

Product

Resources

About