To cite this version:Hedy Attouch, Jérôme Bolte, Benar Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Math. Program., Ser. A, 2011, 137 (1), pp.91-124. <10.1007/s10107-011-0484-9>. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods , 15, 2010; revised: July, 21, 2011 Abstract In view of the minimization of a nonsmooth nonconvex function f , we prove an abstract convergence result for descent methods satisfying a sufficient-decrease assumption, and allowing a relative error tolerance. Our result guarantees the convergence of bounded sequences, under the assumption that the function f satisfies the Kurdyka-Lojasiewicz inequality. This assumption allows to cover a wide range of problems, including nonsmooth semi-algebraic (or more generally tame) minimization. The specialization of our result to different kinds of structured problems provides several new convergence results for inexact versions of the gradient method, the proximal method, the forward-backward splitting algorithm, the gradient projection and some proximal regularization of the Gauss-Seidel method in a nonconvex setting. Our results are illustrated through feasibility problems, or iterative thresholding procedures for compressive sensing.2010 Mathematics Subject Classification: 34G25, 47J25, 47J30, 47J35, 49M15, 49M37, 65K15, 90C25, 90C53.

Abstract. We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: L(x, y) = f (x)+Q(x, y)+g(y), where f : R n → R∪{+∞} and g : R m → R∪{+∞} are proper lower semicontinuous functions, and Q : R n × R m → R is a smooth C 1 function which couples the variables x and y. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize L.We work in a nonconvex setting, just assuming that the function L satisfies the Kurdyka-Lojasiewicz inequality. An entire section illustrates the relevancy of such an assumption by giving examples ranging from semialgebraic geometry to "metrically regular" problems.Our main result can be stated as follows: If L has the Kurdyka-Lojasiewicz property, then each bounded sequence generated by the algorithm converges to a critical point of L. This result is completed by the study of the convergence rate of the algorithm, which depends on the geometrical properties of the function L around its critical points. When specialized to Q(x, y) = x − y 2 and to f , g indicator functions, the algorithm is an alternating projection mehod (a variant of Von Neumann's) that converges for a wide class of sets including semialgebraic and tame sets, transverse smooth manifolds or sets with "regular "intersection. In order to illustrate our results with concrete problems, we provide a convergent proximal reweighted ℓ 1 algorithm for compressive sensing and an application to rank reduction problems.

We study the convergence of the proximal algorithm applied to nonsmooth functions that satisfy the Lojasiewicz inequality around their generalized critical points. Typical examples of functions complying with these conditions are continuous semialgebraic or subanalytic functions. Following Lojasiewicz's original idea, we prove that any bounded sequence generated by the proximal algorithm converges to some generalized critical point. We also obtain convergence rate results which are related to the flatness of the function by means of Lojasiewicz exponents. Apart from the sharp and elliptic cases which yield finite-time or geometric convergence, the decay estimates that are derived are of the type O(k −s), where s ∈ (0, +∞) depends on the flatness of the function.

In a real Hilbert space H, we study the fast convergence properties as t → +∞ of the trajectories of the second-order evolution equationẍ (t) + α tẋ (t) + ∇Φ(x(t)) = 0,where ∇Φ is the gradient of a convex continuously differentiable function Φ : H → R, and α is a positive parameter. In this inertial system, the viscous damping coefficient α t vanishes asymptotically in a moderate way. For α > 3, we show that any trajectory converges weakly to a minimizer of Φ, just assuming that argmin Φ = ∅. The strong convergence is established in various practical situations. These results complement the O(t −2 ) rate of convergence for the values obtained by Su, Boyd and Candès. Time discretization of this system, and some of its variants, provides new fast converging algorithms, expanding the field of rapid methods for structured convex minimization introduced by Nesterov, and further developed by Beck and Teboulle. This study also complements recent advances due to Chambolle and Dossal.3. Relationship with fast numerical optimization methods: As pointed out in [38, Section 2], for α = 3, (1) can be seen as a continuous version of the fast convergent method of Nesterov (see [29,30,31,32]), and its widely used successors, such as the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), studied in [19]. These methods have a convergence rate of Φ(x k ) − min Φ = O(k −2 ), where k is the number of iterations. As for the continuous-time system (1), convergence of the sequences generated by FISTA and related methods has not been established so far. This is a central and long-standing question in the study of numerical optimization methods.The purpose of this research is to establish the convergence of the trajectories satisfying (1), as well as the sequences generated by the corresponding numerical methods with Nesterov-type acceleration. We also complete the study with several stability properties concerning both the continuous-time system and the algorithms.

No abstract

Abstract. The forward-backward algorithm is a powerful tool for solving optimization problems with a additively separable and smooth + nonsmooth structure. In the convex setting, a simple but ingenious acceleration scheme developed by Nesterov has been proved useful to improve the theoretical rate of convergence for the function values from the standard O(k −1 ) down to O(k −2 ). In this short paper, we prove that the rate of convergence of a slight variant of Nesterov's accelerated forward-backward method, which produces convergent sequences, is actually o(k −2 ), rather than O(k −2 ). Our arguments rely on the connection between this algorithm and a second-order differential inclusion with vanishing damping.Final version published at SIOPT.

No abstract

No abstract