Convergence analysis for Kaczmarz-type methods in a Hilbert space framework

Abstract

Using the concept of stable Hilbert space splittings, we provide a unified approach to the convergence analysis for multiplicative Schwarz methods (a version of alternating directions methods), and in particular Kaczmarz-type methods for solving linear systems. We consider both fixed cyclic and randomized ordering strategies, and cover block versions as well. For the classical Kaczmarz method with cyclic ordering for solving general linear systems Ax = b, a new convergence rate estimate in terms of the generalized condition number of A and logarithmically depending on the rank of A is presented.

Stochastic subspace correction methods and fault tolerance

“…For such a random choice of I m , a convergence estimate for the expectation of the squared error in terms of the stability constants of the space splitting has already been announced without proof in [12] (see Theorem 3 in [27] for the argument). We formulate it in a slightly modified setting including weights for convenience.…”

Random reordering in SOR-type methods

“…For non-singular B, i.e., when | · | B becomes a norm and the system has full rank r = n, the outlined idea of proof for Theorem 1 has been carried out in detail in [10]. The changes for singular B are minimal, the proof of (4) for this case can be found in [11,Theorem 4], see also the proof of Part b) of Theorem 4 in Section 3.…”

Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms

“…Due to the better performance in many situations than existing classical iterative algorithms, randomized iterative algorithms for solving a linear system of equations have attracted much attention recently; see, for example, other works and the references therein. In this paper, we consider the randomized Kaczmarz (RK) algorithm, the randomized Gauss–Seidel (RGS) algorithm, the randomized extended Kaczmarz (REK) algorithm, and the randomized extended Gauss–Seidel (REGS) algorithm .…”

“…In this paper, we consider the randomized Kaczmarz (RK) algorithm, the randomized Gauss–Seidel (RGS) algorithm, the randomized extended Kaczmarz (REK) algorithm, and the randomized extended Gauss–Seidel (REGS) algorithm . Convergence rates of these algorithms (including their deterministic variants) have been considered extensively; see, for example, other works . Let A † denote the Moore–Penrose pseudoinverse of A .…”

“…11 Convergence rates of these algorithms (including their deterministic variants) have been considered extensively; see, for example, other works. 1,2,5,6,11,13,[20][21][22][23][24][25][26] Let A † denote the Moore-Penrose pseudoinverse 27 of A. We summarize the convergence of RK, RGS, REK, and REGS in expectation to the Moore-Penrose pseudoinverse solution A † b for all types of linear systems in Table 1.Main contributions.…”