First-Order Algorithms for Convex Optimization with Nonseparable Objective and Coupled Constraints

Gao, Xiang; Zhang, Shu Zhong

doi:10.1007/s40305-016-0131-5

Cited by 51 publications

(52 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing works (e.g., [12,39]) show that in t iterations, the LALM for (6) can generate an O( 1 t )-optimal solutionx, namely, |f (x) − f (x * )| = O( 1 t ) and Ax − b = O( 1 t ). In addition, by smoothing technique, [34] gives a first-order method for the problem (3) and establishes its O( 1 t ) convergence rate result.…”

Section: Main Goalmentioning

confidence: 99%

“…All the designed "hard" instances in this paper are built upon Λ and c given in (12). Two immediate observations regarding (12) and (13) are as follows. First, for any u := (u 1 , .…”

Section: Special Linear Constraintsmentioning

confidence: 99%

“…In this subsection, we study two Krylov subspaces that are associated with the matrix Λ and vector c described in (12). In particular, we consider the Krylov subspaces…”

Section: Krylov Subspacesmentioning

confidence: 99%

“…From Lemma 13, we perform another rotation to the instance and can further show that the generated approximate solution falls into a rotated Krylov subspace. Proposition 2 Given m ≤ n and k < m 2 , let Λ and c be the matrix and vector in (12), and let h ∈ K 0 . Suppose that A and b are respectively a multiple of Λ and c and H ∈ S n + satisfying HK 2s−1 ⊆ K 2s for all s ≤ k 2 .…”

mentioning

confidence: 99%

See 3 more Smart Citations

Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

Ouyang

2019

Math. Program.

103

View full text Add to dashboard Cite

On solving a convex-concave bilinear saddle-point problem (SPP), there have been many works studying the complexity results of first-order methods. These results are all about upper complexity bounds, which can determine at most how many efforts would guarantee a solution of desired accuracy. In this paper, we pursue the opposite direction by deriving lower complexity bounds of first-order methods on large-scale SPPs. Our results apply to the methods whose iterates are in the linear span of past first-order information, as well as more general methods that produce their iterates in an arbitrary manner based on first-order information. We first work on the affinely constrained smooth convex optimization that is a special case of SPP. Different from gradient method on unconstrained problems, we show that first-order methods on affinely constrained problems generally cannot be accelerated from the known convergence rate O(1/t) to O(1/t 2 ), and in addition, O(1/t) is optimal for convex problems. Moreover, we prove that for strongly convex problems, O(1/t 2 ) is the best possible convergence rate, while it is known that gradient methods can have linear convergence on unconstrained problems. Then we extend these results to general SPPs. It turns out that our lower complexity bounds match with several established upper complexity bounds in the literature, and thus they are tight and indicate the optimality of several existing first-order methods.

show abstract

Section: Main Goalmentioning

confidence: 99%

“…All the designed "hard" instances in this paper are built upon Λ and c given in (12). Two immediate observations regarding (12) and (13) are as follows. First, for any u := (u 1 , .…”

Section: Special Linear Constraintsmentioning

confidence: 99%

“…In this subsection, we study two Krylov subspaces that are associated with the matrix Λ and vector c described in (12). In particular, we consider the Krylov subspaces…”

Section: Krylov Subspacesmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

Ouyang

2019

Math. Program.

103

View full text Add to dashboard Cite

show abstract

“…In addition to Linearized ADMM and Chambolle-Pock's algorithm, we also include linearized ALM [29] to solve this example. The result of this test is given in Figure 2, where F (x) = h(x).…”

Section: Convergence Guarantees: Ergodic Vs Non-ergodicmentioning

confidence: 99%

An adaptive primal-dual framework for nonsmooth convex minimization

et al. 2019

View full text Add to dashboard Cite

We propose a new self-adaptive, double-loop smoothing algorithm to solve composite, nonsmooth, and constrained convex optimization problems. Our algorithm is based on Nesterov's smoothing technique via general Bregman distance functions. It selfadaptively selects the number of iterations in the inner loop to achieve a desired complexity bound without requiring the accuracy a priori as in variants of Augmented Lagrangian methods (ALM). We prove O 1 k -convergence rate on the last iterate of the outer sequence for both unconstrained and constrained settings in contrast to ergodic rates which are common in ALM as well as alternating direction method-of-multipliers literature. Compared to existing inexact ALM or quadratic penalty methods, our analysis does not rely on the worst-case bounds of the subproblem solved by the inner loop. Therefore, our algorithm can be viewed as a restarting technique applied to the ASGARD method in [58] but with rigorous theoretical guarantees or as an inexact ALM with explicit inner loop termination rules and adaptive parameters. Our algorithm only requires to initialize the parameters once, and automatically update them during the iteration process without tuning. We illustrate the superiority of our methods via several examples as compared to the state-of-the-art.

show abstract

Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights

Chen

Liu

et al. 2017

Math. Program.

View full text Add to dashboard Cite

In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proximal ADMM converges in point-wise manner to a primal-dual solution pair. Moreover, we apply randomly permuted ADMM (RPADMM) to nonseparable multi-block convex optimization, and prove its expected convergence for a class of nonseparable quadratic programming problems. When the linear constraint vanishes, the 2block proximal ADMM and RPADMM reduce to the 2-block cyclic proximal BCD method and randomly permuted BCD (RPBCD). Our study provides the first iterate convergence result for 2-block cyclic proximal BCD without assuming the boundedness of the iterates. We also theoretically establish the expected iterate convergence result concerning multi-block RPBCD for convex quadratic optimization. In addition, we demonstrate that RPBCD may have a worse convergence rate than cyclic proximal BCD for 2-block convex quadratic minimization problems. Although the results on RPADMM and RPBCD are restricted to quadratic minimization models, they provide some interesting insights: 1) random permutation makes ADMM and BCD more robust for multi-block convex minimization problems; 2) cyclic BCD may outperform RPBCD for "nice" problems, and therefore RPBCD should be applied with caution when solving general convex optimization problems.

show abstract

First-Order Algorithms for Convex Optimization with Nonseparable Objective and Coupled Constraints

Cited by 51 publications

References 24 publications

Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

An adaptive primal-dual framework for nonsmooth convex minimization

Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights

Contact Info

Product

Resources

About