Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions

Abstract

This paper is concerned with convex composite minimization problems in a Hilbert space. In these problems, the objective is the sum of two closed, proper, and convex functions where one is smooth and the other admits a computationally inexpensive proximal operator. We analyze a general family of inertial proximal splitting algorithms (GIPSA) for solving such problems. We establish finiteness of the sum of squared increments of the iterates and optimality of the accumulation points. Weak convergence of the entire sequence then follows if the minimum is attained. Our analysis unifies and extends several previous results.We then focus on ℓ 1 -regularized optimization, which is the ubiquitous special case where the nonsmooth term is the ℓ 1 -norm. For certain parameter choices, GIPSA is amenable to a local analysis for this problem. For these choices we show that GIPSA achieves finite "active manifold identification", i.e. convergence in a finite number of iterations to the optimal support and sign, after which GIPSA reduces to minimizing a local smooth function. Local linear convergence then holds under certain conditions. We determine the rate in terms of the inertia, stepsize, and local curvature. Our local analysis is applicable to certain recent variants of the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), for which we establish active manifold identification and local linear convergence. Our analysis motivates the use of a momentum restart scheme in these FISTA variants to obtain the optimal local linear convergence rate.⋆ This manuscript is related to the preprint arXiv:1502.02281, entitled: "A Lyapunov Analysis of FISTA with Local Linear Convergence for Sparse Optimization". This manuscript is a thoroughly revised and rewritten version which includes several new results.

Convergence rates of inertial splitting schemes for nonconvex composite optimization

“…Furthermore [27] used a different proof technique to the one used here. This same parameter choice has been considered for convex optimization in [24,28], albeit without the sharp convergence rates derived here. In both the convex and nonconvex settings, employing inertia has been found to improve either the convergence rate or the quality of the obtained local minimum in several studies [12,25,23,24].…”

“…For the nonconvex SCAD this is a new result. For 1-regularized least squares, inertial methods have been shown to achieve local linear convergence in [24,31] under additional strict complementarity or restricted strong convexity assumptions. However, our analysis, which is based on the KL inequality, does not explicitly require these additional assumptions, as the objective function always has a KL exponent of 1/2 [19, Lemma 10].…”

Convergence analysis of an inexact inertial Krasnoselskii-Mann algorithm with applications

“…Further, an inertial alternating direction method of multipliers (ADMM) was developed in [24]. Some recent generalization of the iKM (1.2) can be found in [25][26][27][28].…”

Proximal primal–dual best approximation algorithm with memory

“…The overview of primal-dual approaches to solve (P) has been recently proposed in [32]. Some algorithms to solve (1.1) which rely on including x n−1 into the definition of x n+1 were proposed in [3,4,9,15,30,31,35,36,37,38,39,41,42]. They are mostly based on discretizations of the second order differential system related to the problem (1.2).…”

“…This system, called heavy ball with friction, is exploited in order to accelerate convergence. Indeed, the introduction of the inertial term was shown to improve the speed of convergence significantly [30,31].…”