A primal-dual type algorithm with the O(1/t) convergence rate for large scale constrained convex programs

Abstract

This paper considers large scale constrained convex programs, which are usually not solvable by interior point methods or other Newton-type methods due to the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, large scale constrained convex programs are often solved by gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, aka, Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with the O(1/ √ t) convergence rate, where t is the number of iterations. If the objective and constraint functions are separable, the Lagrangian dual type method can decompose a large scale convex program into multiple parallel small scale convex programs. The classical dual gradient algorithm is an example of Lagrangian dual type methods and has convergence rate O(1/ √ t). Recently, a new Lagrangian dual type algorithm with faster O(1/t) convergence is proposed in [1]. However, if the objective or constraint functions are not separable, each iteration of the Lagrangian dual type method in [1] requires to solve a large scale unconstrained convex program, which can have huge complexity. This paper proposes a new primal-dual type algorithm, which only involves simple gradient updates at each iteration and has the O(1/t) convergence rate.

Online First-Order Framework for Robust Convex Optimization

“…Our OFO-based approach aims to improve on this by reducing the per-iteration cost of each step to simple first-order updates. Two exceptions within the primal-dual methods are the work of Nedić and Ozdaglar [33] and Yu and Neely [46], which have cheap per-iteration cost based on only gradient computations and projection operations in the Euclidean setup. Nedić and Ozdaglar [33] provide a convergence rate of O(1/ √ T ) in the non-smooth case.…”

“…In contrast, our framework directly uses the functions f i (x, u i ), so it does not need to take into account the inexact gradient information, and can certify infeasibility of (2). Yu and Neely [46] present a method that can guarantee O(1/T ) convergence when all functions are smooth. However, for RO problems, the constraint functions g i (x) are non-smooth due to the supremum operation, thus their results do not apply to RO.…”

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

“…With sufficiently large proximal parameter that depends on the Lipschitz constants of f i 's, the algorithm converges in O(1/k) ergodic rate. The follow-up paper [38] focuses on smooth constrained convex problems and proposes a linearized variant of the algorithm in [39]. Assuming compactness of the set X , it also establishes O(1/k) ergodic convergence of the linearized method.…”

First-Order Methods for Constrained Convex Programming Based on Linearized Augmented Lagrangian Function

“…This linearized ALM also appears as a special case of the methods in Cui et al (2016), Gao and Zhang (2017), Gao et al (2019), Hong et al (2020), and Xu and Zhang (2018), which perform Gauss-Seidel or randomized BCU to the primal variable in the ALM framework. On smooth nonlinearly constrained convex problems, Yu and Neely (2016) proposes a primal-dual type FOM; see Equation (6.1). Assuming compactness of the constraint set, it establishes O(ε −1 ) iteration complexity result to produce a primal ε solution.…”

“…Assuming compactness of the constraint set, it establishes O(ε −1 ) iteration complexity result to produce a primal ε solution. The method in Yu and Neely (2016) has been generalized to nonlinearly constrained convex composite programs in Yu and Neely (2017) and to convex conic programs in Zhao and Zhu (2018), which also establish O(ε −1 ) iteration complexity result. Along the line of inexact ALM, Ahmadi et al (2016) and Necoara et al (2017) give iteration complexity analysis to convex conic programs.…”