LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems

Abstract

We describe a parallel iterative least squares solver named that is based on random normal projection. computes the min-length solution to minx∈ℝn ‖Ax − b‖2, where A ∈ ℝm × n with m ≫ n or m ≪ n, and where A may be rank-deficient. Tikhonov regularization may also be included. Since A is involved only in matrix-matrix and matrix-vector multiplications, it can be a dense or sparse matrix or a linear operator, and automatically speeds up when A is sparse or a fast linear operator. The preconditioning phase consists of a random normal projection, which is embarrassingly parallel, and a singular value decomposition of size ⌈γ min(m, n)⌉ × min(m, n), where γ is moderately larger than 1, e.g., γ = 2. We prove that the preconditioned system is well-conditioned, with a strong concentration result on the extreme singular values, and hence that the number of iterations is fully predictable when we apply LSQR or the Chebyshev semi-iterative method. As we demonstrate, the Chebyshev method is particularly efficient for solving large problems on clusters with high communication cost. Numerical results show that on a shared-memory machine, is very competitive with LAPACK’s DGELSD and a fast randomized least squares solver called Blendenpik on large dense problems, and it outperforms the least squares solver from SuiteSparseQR on sparse problems without sparsity patterns that can be exploited to reduce fill-in. Further experiments show that scales well on an Amazon Elastic Compute Cloud cluster.

Randomized Extended Kaczmarz for Solving Least Squares

“…LSRN [MSM11] did not perform well under a setup in which no parallelization is allowed as the one used here, so we do not include LSRN's performance. DGELSY uses QR factorization with pivoting and DGELSD uses the singular value decomposition.…”

Coresets-Methods and History: A Theoreticians Design Pattern for Approximation and Streaming Algorithms

“…A parallel least squares regression solver LSRN was developed in [78,97]. An implementation of some of the presented sketching techniques named RaProR was made available for the statistics programming language R [53,54,87].…”

Uniform Sampling for Matrix Approximation

“…To improve stablility and achieve log(1/ǫ) dependence, randomized schemes can be combined with known iterative regression algorithms. These methods only require a constant factor spectral approximation with O(d log d) rows and are addressed in [AMT10,CRT11,CW13,MSM14,RT08].…”