Quantile-Based Random Kaczmarz for corrupted linear systems of equations

Abstract

We consider linear systems Ax = b where A ∈ R m×n consists of normalized rows, a i ℓ 2 = 1, and where up to βm entries of b have been corrupted (possibly by arbitrarily large numbers). Haddock, Needell, Rebrova & Swartworth propose a quantile-based Random Kaczmarz method and show that for certain random matrices A it converges with high likelihood to the true solution. We prove a deterministic version by constructing, for any matrix A, a number β A such that there is convergence for all perturbations with β < β A . Assuming a random matrix heuristic, this proves convergence for tall Gaussian matrices with up to ∼ 0.5% corruption (a number that can likely be improved).

On block accelerations of quantile randomized Kaczmarz for corrupted systems of linear equations

“…See section 2 for the formal description of the algorithm and further discussion, and section 1.4 for all theorem statements. Notably, we do not restrict ourselves to the random matrix setting: as in [Ste21b], we show a general guarantee of linear convergence, with a rate depending on the spectral properties of A and its row submatrices (theorem 1.8).…”

“…On the other hand, independent subgaussian rows are nearly mutually orthogonal with high probability (see, e.g. [Ver18]) and have σ min (A τ ) = O(τ /n) when τ ≫ n. Further discussion in [Ste21b] aids in understanding the relative condition on q, β, and σ 2 q−β,min of theorem 1.7, as well as drawing connections to the random matrix case studied in [Had+22].…”

“…Informally, this results in the intersection subspaces being 'close enough' to individual projection points due to non-trivial angles between the solution hyperplanes for individual equations. The incoherence condition is implicitly needed in the existing non-block quantileRK results [Had+22,Ste21b] to ensure that the quantile statistic is representative. In this work, it also appears in the form of a restricted smallest singular value, discussed below.…”

“…In [Ste21b], Steinerberger sought to generalize the theory behind quantileRK beyond the random matrix setting. Indeed, he distilled the critical controls that the random matrix heuristic provides to conditions on the quantity…”

“…However, such methods often require loading the entire system into memory; a requirement that is frequently impractical or impossible in settings where the system is large-scale, such as those systems arising in medical imaging applications [Hou73]. Recent works [HN18a,Had+22,Ste21b] have introduced novel approaches that in fact require only loading small portions (even single rows) of the system into memory at any time, whilst achieving linear convergence even in the presence of large-or adversarially located-corruptions.…”