An Implicit Representation and Iterative Solution of Randomly Sketched Linear Systems

Abstract

Randomized linear system solvers have become popular as they have the potential to reduce floating point complexity while still achieving desirable convergence rates. One particularly promising class of methods, random sketching solvers, has achieved the best known computational complexity bounds in theory, but is blunted by two practical considerations: there is no clear way of choosing the size of the sketching matrix apriori ; and there is a nontrivial storage cost of the projected system. In this work, we make progress towards addressing these issues by implicitly generating the sketched system and solving it simultaneously through an iterative procedure. As a result, we replace the question of the size of the sketching matrix with determining appropriate stopping criteria; we also avoid the costs of explicitly representing the sketched linear system; and our implicit representation also solves the system at the same time, which controls the per-iteration computational costs. Additionally, our approach allows us to generate a connection between random sketching methods and randomized iterative solvers (e.g., randomized Kaczmarz method). As a consequence, we exploit this connection to (1) produce a stronger, more precise convergence theory for such randomized iterative solvers under arbitrary sampling schemes (i.i.d., adaptive, permutation, dependent, etc.), and (2) improve the rates of convergence of randomized iterative solvers at the expense of a user-determined increases in per-iteration computational and storage costs. We demonstrate these concepts on numerical examples on forty-nine distinct linear systems.Gower and Richtrik, 2015]: the noniterative scheme is simply repeated in order to get better convergence properties. We are not doing this, but rather turning the noniterative scheme into an iterative one.2 We will be more precise about what we refer to as base methods. For now, such methods are exemplified by randomized Kaczmarz [Strohmer and Vershynin, 2009] and randomized Gauss-Seidel [Leventhal and Lewis, 2010].

Convergence of Adaptive, Randomized, Iterative Linear Solvers

“…The key inequality on which our results depend is an extension of Meany's inequality [12], derived in [17,Theorem 4], and stated here.…”

“…By [17,Proposition 1], there exists a π ∈ (0, 1) such that these methods are Exploratory, as long as the much weaker condition,…”

“…Most common procedures (e.g., randomized Kaczmarz variants, randomized Coordinate Descent variants) will readily satisfy this, as well as most randomized vector sketching procedures. However, there can be choices in which this is not true, in which case y k will only converge along a subspace of the row space of M , which we describe in [17] for the i.i.d. case.…”

“…Iterative linear solvers are often preferred for solving large-scale linear systems, as they can take advantage of problem structure such as sparsity or bandedness, require inexpensive floating point operations, and can be readily paired with preconditioning techniques [19, see preface]. While such iterative linear solvers as Conjugate Gradients (CG) and the Generalized Minimal Residual method (GM-RES) are still dominant solvers in practice, randomized row-action [8,1,14,23] and column-action iterative solvers [10,25] have been growing in interest for several reasons: they (usually) require very few floating point operations per iteration [5,3]; they have low-memory footprints [9]; they can readily be composed with randomization techniques to quickly produce approximate solutions [23,10,24,6,11,2,7,17]; they can be used for solving systems constructed in a streaming fashion (e.g., [15]), which supports emerging computing paradigms (e.g., [13]); and, just like the more popular iterative Krylov solvers, they can be parallelized, preconditioned or combined with other linear solvers [20,16,4,18];…”

“…To address this issue, building from our previous results [17], we specify a set of general conditions for such solvers under which we can guarantee convergence with probability one (w.p.1.). Moreover, we are also able to provide a worst case rate of convergence, which generalizes the theory for deterministic solvers [1,14] and complements the specialized mean-squared analyses for certain random solvers [23,10,6,2,7,22].…”