A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Abstract

The Kaczmarz method is a popular iterative scheme for solving large, consistent system of overdetermined linear equations. This method has been widely used in many areas such as reconstruction of CT scanned images, computed tomography and signal processing. In the Kaczmarz method, one cycles through the rows of the linear system and each iteration is formed by projecting the current point to the hyperplane formed by the active row. However, the Kaczmarz method may converge very slowly in practice. The randomized Kaczmarz method (RK) greatly improves the convergence rate of the Kaczmarz method, by using the rows of the coefficient matrix in random order rather than in their given order. An obvious disadvantage of the randomized Kaczmarz method is its probability criterion for selecting the active or working rows in the coefficient matrix. In [Z.Z. BAI, W. WU, On greedy randomized Kaczmarz method for solving large sparse linear systems, SIAM Journal on Scientific Computing, 2018, 40: A592-A606], the authors proposed a new probability criterion that can capture larger entries of the residual vector of the linear system as far as possible at each iteration step, and presented a greedy randomized Kaczmarz method (GRK). However, the greedy Kaczmarz method may suffer from heavily computational cost when the size of the matrix is large, and the overhead will be prohibitively large for big data problems. The contribution of this work is as follows. First, from the probability significance point of view, we present a partially randomized Kaczmarz method, which can reduce the computational overhead needed in greedy randomized Kaczmarz method. Second, based on Chebyshev's law of large numbers and Z-test, we apply a simple sampling approach to the partially randomized Kaczmarz method, and propose a randomized Kaczmarz method with simple random sampling for large linear systems. The convergence of the proposed method is established. Third, we apply the new strategy to the ridge regression problem, and propose a partially randomized Kaczmarz method with simple random sampling for ridge regression. Numerical experiments show numerical behavior of the proposed algorithms, and demonstrate their superiority over many state-of-the-art randomized Kaczmarz methods for large linear systems problems and ridge regression problems.

The randomized circumcentered-reflection iteration method for solving consistent linear equations

“…where i k is chosen with probability proportional to a i 2 2 and λ min (A) denote the minimum eigenvalue of A. Since then, various randomized Kaczmarz method are studied and improved by greedy strategies [2], sampling techniques [5], extrapolating acceleration [17], block version [12,20], averaging [18] and other improvements [6,13,23].…”

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

“…Haddock & Ma [15], Haddock & Needell [18], Jiang, Wu & Jiang [22], Li, Lu & Wang [28], Li & Zhang [29], Nutini, Sepehry, Laradji, Schmidt, Koepke & Virani [40] and the author [43]. The approach proposed in [19] is related to work of Haddock & Needell [16,17] where the Random Kaczmarz method was used to detect corruptions (see also [20]).…”