Convergence rates of inertial splitting schemes for nonconvex composite optimization

Abstract

We study the convergence properties of a general inertial first-order proximal splitting algorithm for solving nonconvex nonsmooth optimization problems. Using the Kurdyka-Łojaziewicz (KL) inequality we establish new convergence rates which apply to several inertial algorithms in the literature. Our basic assumption is that the objective function is semialgebraic, which lends our results broad applicability in the fields of signal processing and machine learning. The convergence rates depend on the exponent of the "desingularizing function" arising in the KL inequality. Depending on this exponent, convergence may be finite, linear, or sublinear and of the form O(k −p ) for p > 1.

Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization

“…Moreover, we transfer local convergence rates depending on the KL exponent of the involved functions to the methods listed above. This result builds on a recent classification of local convergence rates depending on the KL exponent from [33,24] (which extends results from [22]).…”

“…Then, for z 0 = (x 0 , x −1 ) sufficiently close to z * , any algorithm that satisfies (H1) and (H2) and starts at z 0 generates a sequence that remains in a neighborhood of z * , has the finite length property, and converges to a pointz = (x,x) withx ∈ M i=1 S i . Finally, we complement our local convergence result by the convergence rate estimates from [24,33]. Assuming the objective function is semi-algebraic, in [24, Theorems 2 and 4] which build on [22,Theorem 3.4], a list of qualitative convergence rate estimates in terms of the KL-exponent is proved.…”

“…As ϕ (s) = cs θ−1 is non-increasing for θ ∈ [0, 1], we have ϕ (F(z k ) − F(z * )) ≤ ϕ (F(z k+1 ) − F(z * )). The remainder of the proof is identical to [24] starting from [24, Inequality (7)], which yields the rates for (F(z k )) k∈N .…”

“…A similar method was considered in [16] by the same authors. The convergence of a generic multi-step method is proved in [37] (see also [24]). A slightly weakened formulation of the popular accelerated proximal gradient algorithm from convex optimization was analyzed in [36].…”

“…Although the fruitful concept of partial smoothness is very interesting, in this paper, we focus on convergence results that can be inferred directly from the KL property. In the general abstract setting, local convergence rates were analyzed in [22,33] and for inertial methods in [33,24]. More specific local convergence rates can be found in [2,40,3,52,14,19].…”