Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity

Abstract

In a real Hilbert space H, we study the fast convergence properties as t → +∞ of the trajectories of the second-order evolution equationẍ (t) + α tẋ (t) + ∇Φ(x(t)) = 0,where ∇Φ is the gradient of a convex continuously differentiable function Φ : H → R, and α is a positive parameter. In this inertial system, the viscous damping coefficient α t vanishes asymptotically in a moderate way. For α > 3, we show that any trajectory converges weakly to a minimizer of Φ, just assuming that argmin Φ = ∅. The strong convergence is established in various practical situations. These results complement the O(t −2 ) rate of convergence for the values obtained by Su, Boyd and Candès. Time discretization of this system, and some of its variants, provides new fast converging algorithms, expanding the field of rapid methods for structured convex minimization introduced by Nesterov, and further developed by Beck and Teboulle. This study also complements recent advances due to Chambolle and Dossal.3. Relationship with fast numerical optimization methods: As pointed out in [38, Section 2], for α = 3, (1) can be seen as a continuous version of the fast convergent method of Nesterov (see [29,30,31,32]), and its widely used successors, such as the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), studied in [19]. These methods have a convergence rate of Φ(x k ) − min Φ = O(k −2 ), where k is the number of iterations. As for the continuous-time system (1), convergence of the sequences generated by FISTA and related methods has not been established so far. This is a central and long-standing question in the study of numerical optimization methods.The purpose of this research is to establish the convergence of the trajectories satisfying (1), as well as the sequences generated by the corresponding numerical methods with Nesterov-type acceleration. We also complete the study with several stability properties concerning both the continuous-time system and the algorithms.

Inducing Uniform Asymptotic Stability in Non-Autonomous Accelerated Optimization Dynamics via Hybrid Regularization

“…Irrespective of the state space selection, under convexity and suitable smoothness assumptions on f , for p ≥ 2 solutions of (2) minimize the sub-optimality measuref (x 1 (t)) : [7]. Under further conditions on f it can also be established that x(t) converges to x * [12]. However, while this type of convergence results are instrumental for the understanding of system (3), they do not provide information related to the robustness properties of the system under small but persistent time-varying disturbances, which could be of adversarial nature.…”

“…Theorem 3.1: Suppose that Assumption 3.1 holds and consider the HAND-1. Then, the following holds: (a) Every maximal solution is complete and the set A, given by (12), is UGAS. (b) For each δ ∈ R >0 and each compact set K 0 ⊂ R 2n there exists an ε * ∈ R >0 and a T ∈ R >0 such that for every perturbation e(t) satisfying sup t |e(t)| ≤ ε * and every initial condition z(0, 0) ∈ K 0 × [T min , T max ] the solutions of the perturbed dynamics (9) satisfy |z(t, j)| A ≤ δ for all (t, j) ∈ dom(z) such that t + j ≥ T .…”

Accelerating Two Projection Methods via Perturbations with Application to Intensity-Modulated Radiation Therapy

“…For a deeper discussion of this matter, see e.g. [1,28] as well as [21,22] and the many references therein.…”

“…Incorporating previous gradient information alleviates zigzagging behavior compared to methods, which only use current gradient information. In recent years authors demonstrated that many optimization algorithms of this kind can be seen as a discretization of the trajectories of differential equations derived from the field of continuous dynamical systems, e.g., [1,24,28,31]. It is shown that inertial approaches are very effective and demonstrate good convergence properties.…”

A new class of accelerated regularization methods, with application to bioluminescence tomography

“…The dominant iterative regularization method for solving (1) should be the Landweber method, given by f δ k+1 = f δ k + ∆tK * (y δ − Kf δ k ), ∆t ∈ (0, 2/ K * K ) (k = 0, 1, 2...) (2) with some starting element f 0 ∈ Q, where K * denotes the adjoint operator of K. The continuous analog to (2) as ∆t tends to zero is known as asymptotic regularization or Showalter's method (see, e.g., [24,25]). It is in the form of a first order evolution equatioṅ…”