Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

Abstract

On solving a convex-concave bilinear saddle-point problem (SPP), there have been many works studying the complexity results of first-order methods. These results are all about upper complexity bounds, which can determine at most how many efforts would guarantee a solution of desired accuracy. In this paper, we pursue the opposite direction by deriving lower complexity bounds of first-order methods on large-scale SPPs. Our results apply to the methods whose iterates are in the linear span of past first-order information, as well as more general methods that produce their iterates in an arbitrary manner based on first-order information. We first work on the affinely constrained smooth convex optimization that is a special case of SPP. Different from gradient method on unconstrained problems, we show that first-order methods on affinely constrained problems generally cannot be accelerated from the known convergence rate O(1/t) to O(1/t 2 ), and in addition, O(1/t) is optimal for convex problems. Moreover, we prove that for strongly convex problems, O(1/t 2 ) is the best possible convergence rate, while it is known that gradient methods can have linear convergence on unconstrained problems. Then we extend these results to general SPPs. It turns out that our lower complexity bounds match with several established upper complexity bounds in the literature, and thus they are tight and indicate the optimality of several existing first-order methods.

First-Order Methods for Constrained Convex Programming Based on Linearized Augmented Lagrangian Function

“…The recent work of Hamedani and Aybat (2019) proposes distributed primal-dual FOMs for strongly convex conic constrained problems on either static or time-varying networks. The algorithms need O(ε − 1 2 ) iterations to achieve a primal ε solution, and the result matches with a lower complexity bound in Ouyang and Xu (2019) for bilinear SP problems. Another line of research is the level-set method of Aravkin et al (2019) and Lin et al (2018), which can also apply FOMs to each subproblem.…”

Some worst-case datasets of deterministic first-order methods for solving binary logistic regression

“…A few remarks are in place regarding the above construction of A k in equation (2.2). First, the construction of the symmetric matrix W k follows the worst-case instance of convex-concave saddle-point problems in [11], which is a slight modification of Nesterov's tridiagonal worst-case matrix for convex quadratic programming [10]. Indeed, W 2 k yields a tridiagonal matrix differs from Nesterov's construction in [10] by only one entry.…”

“…In this section, we extend the lower complexity bound to general deterministic firstorder methods. The derivation is based on the concept of orthogonal invariance in the seminal work of [9], and is organized in a similar way as in [11]. Note that we can also use the concept of zero-respecting algorithms in [2,3] to finish the proof.…”

“…Note that we can also use the concept of zero-respecting algorithms in [2,3] to finish the proof. We will use the following technical lemma which is proved in [11] (see Lemma 3.1 within). Lemma 3.1.…”

“…• When f is weakly smooth convex with parameter ρ (see the definition of the class of weakly smooth functions in [7]), the lower complexity bound is O(1)(1/ε 2/(1+3ρ) ) [7,6]. • When f is convex nonsmooth with bilinear saddle point structure, the lower complexity bound is O(1)(1/ε) [11]. • When f is smooth convex, the lower complexity bound is O(1)(1/ √ ε) [9,10,6,5,12,2,3,4].…”