Balancing Communication and Computation in Distributed Optimization

Abstract

Methods for distributed optimization have received significant attention in recent years owing to their wide applicability in various domains including machine learning, robotics and sensor networks. A distributed optimization method typically consists of two key components: communication and computation. More specifically, at every iteration (or every several iterations) of a distributed algorithm, each node in the network requires some form of information exchange with its neighboring nodes (communication) and the computation step related to a (sub)-gradient (computation). The standard way of judging an algorithm via only the number of iterations overlooks the complexity associated with each iteration. Moreover, various applications deploying distributed methods may prefer a different composition of communication and computation.Motivated by this discrepancy, in this work we propose an adaptive cost framework which adjusts the cost measure depending on the features of various applications. We present a flexible algorithmic framework, where communication and computation steps are explicitly decomposed to enable algorithm customization for various applications. We apply this framework to the well-known distributed gradient descent (DGD) method, and show that the resulting customized algorithms, which we call DGD t , NEAR-DGD t and NEAR-DGD + , compare favorably to their base algorithms, both theoretically and empirically. The proposed NEAR-DGD + algorithm is an exact first-order method where the communication and computation steps are nested, and when the number of communication steps is adaptively increased, the method converges to the optimal solution. We test the performance and illustrate the flexibility of the methods, as well as practical variants, on quadratic functions and classification problems that arise in machine learning, in terms of iterations, gradient evaluations, communications and the proposed cost framework.

Nested Distributed Gradient Methods with Stochastic Computation Errors

“…We can now begin to derive the convergence properties of the stochastic NEAR-DGD t method. We will closely follow the analysis in [9]. We start by proving that the magnitude of the system-wide stochastic NEAR-DGD t iterates is upper bounded in expectation.…”

“…We will now briefly summarize the NEAR-DGD method first published in [9]. Each iteration of NEAR-DGD is composed of a number of successive consensus steps, during which agents communicate with their neighbors, followed by a gradient step executed locally.…”

“…A class of Nested Distributed Gradient methods (NEAR-DGD) was first introduced in [9]. The power of NEAR-DGD lies in its versatile framework, which alternates between gradient steps and an adjustable number of nested consensus steps.…”

A General Framework of Exact Primal-Dual First-Order Algorithms for Distributed Optimization

“…., x n ] ∈ R n , and Ax = 0 represents all equality constraints. Some choices for matrix A include edge-node incidence matrix [4], weighted incidence matrix [45], graph Laplacian matrix [41], and weighted Laplacian matrix [23,1]. In this paper, we choose matrix A to be the edge-node incidence matrix of the network graph, i.e., A ∈ R ×n , = |E|, whose null space is spanned by the vector of all ones.…”

“…We simulate FlexPD-C algorithm and use its theoretical bounds for stepsize. The objective function at each agent i is of the form f i (x) = c i (x i − b i ) 2 with c i and b i being integers that are randomly chosen from [1, 10 3 ] and [1,100]. We run the simulation for 1000 random seeds.…”