Parallel and distributed sparse optimization

Abstract

This paper proposes parallel and distributed algorithms for solving very large-scale sparse optimization problems on computer clusters and clouds. Modern datasets usually have a large number of features or training samples, and they are usually stored in a distributed manner. Motivated by the need of solving sparse optimization problems with large datasets, we propose two approaches including (i) distributed implementations of prox-linear algorithms and (ii) GRock, a parallel greedy block coordinate descent method. Different separability properties of the objective terms in the problem enable different data distributed schemes along with their corresponding algorithm implementations. We also establish the convergence of GRock and explain why it often performs exceptionally well for sparse optimization. Numerical results on a computer cluster and Amazon EC2 demonstrate the efficiency and elasticity of our algorithms.

A novel line search method for nonsmooth optimization problems

“…t } generated by (5) with the simplified exact line search (12) or the simplifed successive line search (14) is a stationary point of (1).…”

“…Thus the proposed line search methods (12) and (14) generally yields faster convergence than state-of-theart decreasing stepsizes (6) and are easier to implement than state-of-the-art line search techniques (8) and (9), as will be illustrated in the next section for the example of the LASSO problem in sparse signal estimation.…”

“…In this section, we specialize the proposed algorithm to solve the LASSO problem, an important and widely studied problem in sparse signal estimation [4,[8][9][10][11][12]:…”

Fault Location Using Wide-Area Measurements and Sparse Estimation

“…Thus investigations are made on how the sparse estimation problem can be decomposed allowing for solution using multiple processors running in parallel. Based on [40], there can be three types of data distribution for the dictionary matrix namely row block distribution, column block distribution and general block distribution are illustrated, as shown in Fig. 4.2 below.…”

Distributed Coordinate Descent for L1-regularized Logistic Regression

“…They present theoretical estimates of speed-up factor of parallelization. Peng et al [12] proposed a greedy block-coordinate descent method, which selects the next coordinate to update based on the estimate of the expected improvement in the objective. They found their GRock algorithm to be superior over parallel FISTA and ADMM.…”