Greedy and randomized versions of the multiplicative Schwarz method

Abstract

We consider sequential, i.e., Gauss-Seidel type, subspace correction methods for the iterative solution of symmetric positive definite variational problems, where the order of subspace correction steps is not deterministically fixed as in standard multiplicative Schwarz methods. Here, we greedily choose the subspace with the largest (or at least a relatively large) residual norm for the next update step, which is also known as the Gauss-Southwell method. We prove exponential convergence in the energy norm, with a reduction factor per iteration step directly related to the spectral properties, e.g., the condition number, of the underlying space splitting. To avoid the additional computational cost associated with the greedy pick, we alternatively consider choosing the next subspace randomly, and show similar estimates for the expected error reduction. We give some numerical examples, in particular applications to a Toeplitz system and to multilevel discretizations of an elliptic boundary value problem, which illustrate the theoretical estimates

A Sampling Kaczmarz--Motzkin Algorithm for Linear Feasibility

“…The relation of these approaches to coordinate descent and gradient descent methods has also been recently studied, see e.g. [GO12,Dum14,NSW14a [LW15,LMY15], and the use of preconditioning [GPS16]. Some other references on recent work include [CP12,RM12] For the most part, it seems that these two branches of research which address the same problems have been developing disjointly from each other.…”

Convergence analysis for Kaczmarz-type methods in a Hilbert space framework

“…This paper is a reaction to a number of recent publications [1][2][3][4][5] on randomized versions of the Kaczmarz method triggered by Strohmer and Vershynin [6], and should be viewed as an addendum to [7,8]. The latter two papers are devoted to the theory of so-called Schwarz iterative (or subspace correction) methods for solving elliptic variational problems in Hilbert spaces.…”

“…The latter two papers are devoted to the theory of so-called Schwarz iterative (or subspace correction) methods for solving elliptic variational problems in Hilbert spaces. That the Kaczmarz method is a particular instance of Schwarz iterative methods has been pointed out in [8]. Alternatively, the Kaczmarz method is a special case of the Neumann-Halperin alternating directions method (ADM) for finding a point in the intersection of many (affine) subspaces of a Hilbert space [9,10] which in turn is part of the family of projection onto convex sets (POCS) algorithms that is popular in many applications (e.g.…”

“…This framework is briefly introduced in Section 2. It leads to generic upper bounds for the convergence speed of Schwarz iterative methods (and thus ADM and, in particular, Kaczmarz methods) for deterministic cyclic [7], greedy, and random orderings [8] in terms of the spectral properties of a transformed operator equation generated by the original problem and its splitting into subproblems. Since the convergence estimate for cyclic orderings obtained in [7] was not proved in full generality, and does not seem to appear in the ADM and POCS literature, we state it here as Theorem 1, and give a short proof of it.…”

Fast iterative graph computation with block updates

“…However, block-level scheduling results in less scheduling overhead and better cache utilization, and researchers have successfully used block scheduling with block priorities defined by aggregation in several applications [8,13,41]. Our framework is general enough to support all the aggregation methods used in these papers.…”