$\ell_1$ Regression using Lewis Weights Preconditioning and Stochastic Gradient Descent

Abstract

We present preconditioned stochastic gradient descent (SGD) algorithms for the ℓ 1 minimization problem min x Ax − b 1 in the overdetermined case, where there are far more constraints than variables. Specifically, we have A ∈ R n×d for n ≫ d. Commonly known as the Least Absolute Deviations problem, ℓ 1 regression can be used to solve many important combinatorial problems, such as minimum cut and shortest path. SGD-based algorithms are appealing for their simplicity and practical efficiency. Our primary insight is that careful preprocessing can yield preconditioned matrices à with strong properties (besides good condition number and low-dimension) that allow for faster convergence of gradient descent. In particular, we precondition using Lewis weights to obtain an isotropic matrix with fewer rows and strong upper bounds on all row norms. We leverage these conditions to find a good initialization, which we use along with recent smoothing reductions and accelerated stochastic gradient descent algorithms to achieve ǫ relative error in O(nnz(A) + d 2.5 ǫ −2 ) time with high probability, where nnz(A) is the number of non-zeros in A. This improves over the previous best result using gradient descent for ℓ 1 regression. We also match the best known running times for interior point methods in several settings.Finally, we also show that if our original matrix A is approximately isotropic and the row norms are approximately equal, we can give an algorithm that avoids using fast matrix multiplication and obtains a running time of O(nnz(A)+sd 1.5 ǫ −2 +d 2 ǫ −2 ), where s is the maximum number of nonzeros in a row of A. In this setting, we beat the best interior point methods for certain parameter regimes.

The Communication Complexity of Optimization

“…On the other hand, the complexity of first-order methods usually has dependences on various parameters of the input matrix A, which can be unbounded in the worst case. Fortunately, recent developments in ℓ 1 regression [32] show how to precondition the matrix A by simply doing an ℓ 1 Lewis weights sampling, and then rotating the matrix appropriately. By carefully combining this preconditioning procedure with Accelerated Gradient Descent, we obtain an algorithm for (1+ε)-approximate ℓ 1 regression with communication complexity O(sd 3 L/ε) in the coordinator model, which shows it is indeed possible to improve the ε dependence for ℓ 1 regression.…”

“…), we use Lemma 29 in [32], which states that with constant probability, the leverage scores of SA satisfy τ i (SA) = O(d/N ) for all i. Since leverage scores are invariant under change of basis (see, e.g., Section 2.4 in [65]), we have for all i,…”

“…where we assume SAR −1 x − Sb 1 is G-Lipschitz continuous and the initial solution x satisfies x − x * 2 2 ≤ Θ. Since SAR −1 is approximately isotropic row-bounded and the initial vector x is the optimal solution to the ℓ 2 regression problem min x SAR −1 x − Sb 2 2 , Lemma 19 in [32] shows that x − x * 2 ≤ d/n Ax * − b 1 . Furthermore, Lemma 15 in [32] shows that G ≤ √ nd.…”

“…Since SAR −1 is approximately isotropic row-bounded and the initial vector x is the optimal solution to the ℓ 2 regression problem min x SAR −1 x − Sb 2 2 , Lemma 19 in [32] shows that x − x * 2 ≤ d/n Ax * − b 1 . Furthermore, Lemma 15 in [32] shows that G ≤ √ nd. By setting δ = ε Ax * − b 1 , we can calculate a (1 + ε)-approximate solution to the ℓ 1 regression problem using O(d/ε) full gradient calculations.…”