Faster randomized block sparse Kaczmarz by averaging

Abstract

The standard randomized sparse Kaczmarz (RSK) method is an algorithm to compute sparse solutions of linear systems of equations and uses sequential updates, and thus, does not take advantage of parallel computations. In this work, we introduce a parallel (mini batch) version of RSK based on averaging several Kaczmarz steps. Naturally, this method allows for parallelization and we show that it can also leverage large overrelaxation. We prove linear expected convergence and show that, given that parallel computations can be exploited, the method provably provides faster convergence than the standard method. This method can also be viewed as a variant of the linearized Bregman algorithm, a randomized dual block coordinate descent update, a stochastic mirror descent update, or a relaxed version of RSK and we recover the standard RSK method when the batch size is equal to one. We also provide estimates for inconsistent systems and show that the iterates converges to an error in the order of the noise level. Finally, numerical examples illustrate the benefits of the new algorithm.

A randomized sparse Kaczmarz solver for sparse signal recovery via minimax‐concave penalty

“…Inspired by the works of Petra [34] and Tondji et al [35], we show that the sparse Kaczmarz algorithm is a particular instance of the coordinate descent method applied to an unconstrained dual problem corresponding to a regularized MCP problem (8). Since the conjugate of the function F 𝛿 (x) is…”

“…Inspired by the works of Petra [34] and Tondji et al [35], we show that the sparse Kaczmarz algorithm is a particular instance of the coordinate descent method applied to an unconstrained dual problem corresponding to a regularized MCP problem (). Since the conjugate of the function $$$ {F}_{\delta }(x) $$$ is $$$$ {\displaystyle \begin{array}{cc}\hfill {F}_{\delta}^{\ast }(z)=& \kern0.2em \underset{x\in {\mathrm{\mathbb{R}}}^n}{\max}\left(\left\langle z,x\right\rangle -{F}_{\delta }(x)\right)\hfill \\ {}\hfill =& \kern0.2em \sum \limits_{i=1}^n\underset{x_i\in \mathrm{\mathbb{R}}}{\max}\left({z}_i{x}_i-\lambda {l}_{\delta}\left({x}_i\right)-\frac{1}{2}{x}_i^2\right)\hfill \\ {}\hfill =& \kern0.2em \sum \limits_{i=1}^n{f}_{\delta}^{\ast}\left({z}_i\right),\hfill \end{array}} $$$$ where …”

A weighted randomized sparse Kaczmarz method for solving linear systems

“…We would like to thank Dr. Lionel N. Tondji et.al. for sending us their conference paper Tondji et al (2021) to kindly remind us that they also independently proposed the weighted randomized sparse Kaczmarz method. This work was supported by the National Natural Science Foundation of China (No.11971480, No.61977065), the Natural Science Fund of Hunan for Excellent Youth (No.2020JJ3038), and the Fund for NUDT Young Innovator Awards (No.20190105).…”

Stochastic mirror descent method for linear ill-posed problems in Banach spaces

“…There exist extensive studies on the mirror descent method and its stochastic variants in optimization, see [6,9,30,31,40,42] for instance. The existing works either depend crucially on the finite-dimensionality of the underlying spaces or establish only error estimates in terms of objective function values, and therefore they are not applicable to our algorithm 1 for illposed problems.…”