Stochastic Dual Ascent for Solving Linear Systems

Abstract

We develop a new randomized iterative algorithm-stochastic dual ascent (SDA)-for finding the projection of a given vector onto the solution space of a linear system. The method is dual in nature: with the dual being a non-strongly concave quadratic maximization problem without constraints. In each iteration of SDA, a dual variable is updated by a carefully chosen point in a subspace spanned by the columns of a random matrix drawn independently from a fixed distribution. The distribution plays the role of a parameter of the method. Our complexity results hold for a wide family of distributions of random matrices, which opens the possibility to fine-tune the stochasticity of the method to particular applications. We prove that primal iterates associated with the dual process converge to the projection exponentially fast in expectation, and give a formula and an insightful lower bound for the convergence rate. We also prove that the same rate applies to dual function values, primal function values and the duality gap. Unlike traditional iterative methods, SDA converges under no additional assumptions on the system (e.g., rank, diagonal dominance) beyond consistency. In fact, our lower bound improves as the rank of the system matrix drops. Many existing randomized methods for linear systems arise as special cases of SDA, including randomized Kaczmarz, randomized Newton, randomized coordinate descent, Gaussian descent, and their variants. In special cases where our method specializes to a known algorithm, we either recover the best known rates, or improve upon them. Finally, we show that the framework can be applied to the distributed average consensus problem to obtain an array of new algorithms. The randomized gossip algorithm arises as a special case.

Convergence Analysis of Inexact Randomized Iterative Methods

“…A direct dual method for solving problem (10) was first proposed in [19]. The dual method-Stochastic Dual Subspace Ascent (SDSA)-updates the dual vectors y k as follows:…”

“…It can be proved, [19,29], that the iterates {x k } k≥0 of the sketch and project method (8) arise as affine images of the iterates {y k } k≥0 of the dual method (11) as follows:…”

“…In [19] the dual method was analyzed for the case of unit stepsize (ω = 1). Later in [29] the analysis extended to capture the cases of ω ∈ (0, 2).…”

“…An interesting property that holds between the suboptimalities of the Sketch and Project method and SDSA is that the dual suboptimality of y in terms of the dual function values is equal to the primal suboptimality of x(y) in terms of distance [19,29]. That is,…”

Preconvergence of the randomized extended Kaczmarz method

“…If the system (1) is consistent, Strohmer and Vershynin [10] proved the linear convergence of the randomized Kaczmarz (RK) method for overdetermined full rank linear system. In fact, the RK method has the same convergence property regardless of whether the system is overdetermined or underdetermined, full rank or rank deficient; see [11,12] for more details. Now, the RK method has been extended to solve various problems including linear constraint problem [13], ridge regression problem [14,15], linear feasibility problem [16], generalized phase retrieval problem [17], and inverse problem [18], and has many variants [19][20][21][22][23][24].…”