A Closed-Form Bound on the Asymptotic Linear Convergence of Iterative Methods via Fixed Point Analysis

Abstract

We present a tight closed-form bound on the convergence of the quadratic first-order difference equation with positive coefficients. Our result offers insight into the convergence behavior of the dynamic system in its convergent regime: while the quadratic term is negligible in the asymptotic regime, it leads to a finite increase in the number of iterations required for a given accuracy relative to the corresponding linear difference equation.

On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares

“…Next, applying Theorem 1 in [49], under the condition a 0 = δ(0) < (1 − ρ(H))/(q/ Q −1 2 ), yields a k ≤ ǫa 0 for any integer k satisfies (30). From (91), we further have δ(k) ≤ a k ≤ ǫa 0 = ǫ δ(0) .…”

“…to establish an asymptotically-linear quadratic system dynamic that upper-bounds the norm of the transformed error vector, (3) applying the result on the convergence of an asymptotically-linear quadratic difference equation in [49] to obtain the number of iterations required for δ(k) ≤ ǫ δ(0) , and (4) converting the convergence result on the transformed error δ(k) to the convergence result on the original error δ (k) . In the following, we provide the complete proof, with some details deferred to the appendix.…”

“…Step 3: If we replace the inequality symbol in ( 29) by the equality symbol, then we obtain an asymptotically-linear quadratic difference equation whose convergence is studied in [49]. Indeed, the following lemma states that the norm of the transformed error vector is governed by this asymptoticallylinear quadratic system dynamic:…”

“…APPENDIX F PROOF OF LEMMA 4 In this section, we show the convergence of { δ(k) } ∞ k=0 using Theorem 1 in [49]. Our idea is to consider a surrogate sequence {a k } ∞ k=0 that upper-bounds { δ(k) } ∞ k=0 :…”

Local Convergence of the Heavy Ball Method in Iterative Hard Thresholding for Low-rank Matrix Completion

“…The nonlinear difference equation ( 13) has been well-studied in the stability theory of difference equations [41]- [43]. In fact, our theorem follows directly on applying Theorem 1 in [41] to (13), with a 0 = E (0) F , ρ = 1 − λ min (H), and q = c 1 /σ r . The proofs of Lemmas 1 and 2 are given in Appendix A.…”