Conditions for linear convergence of the gradient method for non-convex optimization

Abstract

In this paper, we derive a new linear convergence rate for the gradient method with fixed step lengths for non-convex smooth optimization problems satisfying the Polyak-Lojasiewicz (P L) inequality. We establish that the P L inequality is a necessary and sufficient condition for linear convergence to the optimal value for this class of problems. We list some related classes of functions for which the gradient method may enjoy linear convergence rate. Moreover, we investigate their relationship with the P L inequality.

The exact worst-case convergence rate of the alternating direction method of multipliers

“…Recently the authors showed that the P L inequality is necessary and sufficient conditions for the linear convergence of the gradient method with constant step lengths for L-smooth function; see [2,Theorem 5]. In what follows, we establish that the P L inequality is a necessary condition for the linear convergence of ADMM.…”

“…Deng et al [7] show that the sequence {(x k , z k , λ k )} is convergent linearly to a saddle point under the two scenarios given in Table 1. It is worth mentioning that Scenario 1 or Scenario 2 implies strong convexity of the dual objective function and therefore the P L inequality is resulted, see [2]. Hence, Theorem 6 implies the linear convergence in terms of dual value under Scenario 1 or Scenario 2.…”

“…Recently, some scholars established the linear convergence of gradient methods for L-smooth convex functions by replacing strong convexity with some mild conditions, see [2,4,30] and references therein. Inspired by these results, we prove the linear convergence of ADMM by using the so-called P L inequality.…”

Optimal first-order methods for convex functions with a quadratic upper bound

“…We verify that x 1 = − 1 4 M γ and that the sequence (x t ) t cycles back to 3 4 M γ. Therefore, the sequence itself does not converge and the sequence of the PR averaged iterates converges to 1 4 M γ, while the optimum value would be 0.…”

“…Therefore, many authors studied conditions in between convexity and strong convexity, aiming to obtain faster rates under relatively generic assumptions. In particular, different authors considered the restricted secant inequality [55,22], the error bound [35], Łojasiewicz-type inequalities [44], and many more [25,29,33,19,36,24,1].…”