Distributed Solution of Large-Scale Linear Systems via Accelerated Projection-Based Consensus

Abstract

Solving a large-scale system of linear equations is a key step at the heart of many algorithms in machine learning, scientific computing, and beyond. When the problem dimension is large, computational and/or memory constraints make it desirable, or even necessary, to perform the task in a distributed fashion. In this paper, we consider a common scenario in which a taskmaster intends to solve a large-scale system of linear equations by distributing subsets of the equations among a number of computing machines/cores. We propose an accelerated distributed consensus algorithm, in which at each iteration every machine updates its solution by adding a scaled version of the projection of an error signal onto the nullspace of its system of equations, and where the taskmaster conducts an averaging over the solutions with momentum. The convergence behavior of the proposed algorithm is analyzed in detail and analytically shown to compare favorably with the convergence rate of alternative distributed methods, namely distributed gradient descent, distributed versions of Nesterov's accelerated gradient descent and heavy-ball method, the block Cimmino method, and ADMM. On randomly chosen linear systems, as well as on real-world data sets, the proposed method offers significant speed-up relative to all the aforementioned methods. Finally, our analysis suggests a novel variation of the distributed heavy-ball method, which employs a particular distributed preconditioning, and which achieves the same theoretical convergence rate as the proposed consensus-based method.in engineering and the sciences. In particular, we consider the setting in which a taskmaster intends to solve a large-scale system of equations with the help of a set of computing machines/cores (Figure 1). This problem can in general be cast as an optimization problem with a cost function that is separable in the data (but not in the variables) 1 . Hence, there are general approaches to construct distributed algorithms for this problem, such as distributed versions of gradient descent and its variants (e.g. Nesterov's accelerated gradient [15], heavy-ball method [16], etc.), where each machine computes the partial gradient corresponding to a term in the cost and the taskmaster then aggregates the partial gradients by summing them, as well as the so-called Alternating Direction Method of Multipliers (ADMM) and its variants [3]. Among others, some recent approaches for Distributed Gradient Descent (DGD) have been presented and analyzed in [23], [17] and [21], and also coding techniques for robust DGD in the presence of failures and straggler machines have been studied in [10,20]. ADMM has been widely used [7,5,22] for solving various convex optimization problems in a distributed way, and in particular for consensus optimization [13,18,12], which is the relevant one for the type of separation that we have here.

Robustness of Iteratively Pre-Conditioned Gradient-Descent Method: The Case of Distributed Linear Regression Problem

“…Under Assumption 1, A T A is invertible, and we let K * = A T A −1 . Note, from (1), that the Hessian of the aggregate cost function m i=1 F i (x) is equal to A T A for all x ∈ R d . Thus, under Assumption 1 when A T A is positive definite, the aggregate cost function has a unique minimum point.…”

“…This paper considers the problem of multi-agent distributed linear regression in the presence of additive system noises, namely the observation noise and process noise. The nomenclature distributed refers to the data being distributed amongst multiple agents [1,2]. In this problem, the goal is to design an algorithm that -…”

“…There are numerous distributed algorithms available for solving the above regression problem, for example, see [1,9] and references therein. A few notable ones are the gradient-descent (GD) method [2], Nesterov's accelerated gradient-descent method (NAG) [10], the heavy-ball method (HBM) [11], the accelerated projection-consensus method (APC) [1], the quasi-Newton BFGS method [12], and a recently proposed iteratively pre-conditioned gradient-descent (IPG) method [13]. These distributed algorithms are iterative wherein the server maintains an estimate of a solution defined by (2), which is updated iteratively using the gradients of the individual agents' local cost functions defined in (1).…”

“…A few notable ones are the gradient-descent (GD) method [2], Nesterov's accelerated gradient-descent method (NAG) [10], the heavy-ball method (HBM) [11], the accelerated projection-consensus method (APC) [1], the quasi-Newton BFGS method [12], and a recently proposed iteratively pre-conditioned gradient-descent (IPG) method [13]. These distributed algorithms are iterative wherein the server maintains an estimate of a solution defined by (2), which is updated iteratively using the gradients of the individual agents' local cost functions defined in (1). In an ideal scenario with no noise, these algorithms converge to an optimal regression parameter defined in (2).…”

Distributed Weighted Least-Squares and Gaussian Belief Propagation: An Integrated Approach

“…With this approach, a global solution of the least squares problem is obtained via the agreement reached through the communication between neighbouring subsystems [18]. In [19] authors present accelerated projection-based consensus algorithm in which all subsystems, interested in reaching consensus, communicate to a central coordinator, which averages all current estimates to obtain globally optimal estimates. Authors in [10] propose a distributed algorithm for solving WLS estimation problem, where every subsystem aims to estimate one component of the state variable in a globally optimal sense.…”