Communication-Censored Linearized ADMM for Decentralized Consensus Optimization

Abstract

In this paper, we propose a communication-and computation-efficient algorithm to solve a convex consensus optimization problem defined over a decentralized network. A remarkable existing algorithm to solve this problem is the alternating direction method of multipliers (ADMM), in which at every iteration every node updates its local variable through combining neighboring variables and solving an optimization subproblem. The proposed algorithm, called as communicationcensored linearized ADMM (COLA), leverages a linearization technique to reduce the iteration-wise computation cost of ADMM and uses a communication-censoring strategy to alleviate the communication cost. To be specific, COLA introduces successive linearization approximations to the local cost functions such that the resultant computation is first-order and light-weight. Since the linearization technique slows down the convergence speed, COLA further adopts the communicationcensoring strategy to avoid transmissions of less informative messages. A node is allowed to transmit only if the distance between the current local variable and its previously transmitted one is larger than a censoring threshold. COLA is proven to be convergent when the local cost functions have Lipschitz continuous gradients and the censoring threshold is summable. When the local cost functions are further strongly convex, we establish the linear (sublinear) convergence rate of COLA, given that the censoring threshold linearly (sublinearly) decays to 0. Numerical experiments corroborate with the theoretical findings and demonstrate the satisfactory communication-computation tradeoff of COLA. . 2 that approximates the second-order information with local gradients. The lower complexity bounds and rate-optimal algorithms of decentralized optimization are developed in [36]- [38]. Note that the communication cost in the aforementioned algorithms is proportional to the number of iterations, since after a given number of iterations every node needs to communicate with its neighbors.In all decentralized algorithms, there is an essential communication-computation tradeoff [39]- [43]. An algorithm with light-weight iteration-wise computation generally needs more number of iterations, and in consequence more communication cost, to reach a target accuracy. For example, compared with ADMM, DLM enjoys simple gradient-based computation, but suffers from relatively slow convergence speed and high communication cost. In this paper, we aim at achieving a favorable communication-computation tradeoff in a decentralized network, where the nodes are only affordable to light-weight gradient-based computation. The limitation on the computation power may come from that the nodes are equipped with cheap computing units in a wireless sensor network, or from that using higher-order information is prohibitively time-consuming for finding a high-dimensional solution in a machine learning system.Given the constraint on the computation cost, we adopt the communication-censoring strategy to further save the commu...

Coded Stochastic ADMM for Decentralized Consensus Optimization With Edge Computing

“…To investigate the communication efficiency, we compare our approaches with state-of-the-art consensus optimization methods: 1) WADMM in [5], where the agent activating order follows a random walk over the network; 2) D-ADMM in [16]; 3) DGD in [8]; and 4) EXTRA in [9], with respect to the accuracy [11], which is defined as…”

“…In [5]- [11], a few distributed algorithms have been developed to address optimization problem (1). Currently, primal and primal-dual methods are two main widely used solutions, which include, e.g., gradient descent (GD)-based methods and alternating direction method of multipliers (ADMMs)-based methods, respectively.…”

“…Thus, alternative techniques, in particular, the distributed ADMM (D-ADMM) in [16], the communication-censored ADMM (COCA) in [11], and Group ADMM (GADMM) in [17], have been proposed to limit the overall communication load in each iteration. Specifically, for reducing communication costs, eliminating less informative message sharing is preferred.…”

“…Specifically, for reducing communication costs, eliminating less informative message sharing is preferred. In [11], the proposed COCA was able to adaptively determine whether or not a message is informative during the optimization process. Following COCA, communicationcensored linearized ADMM (COLA) was introduced in [18] to take into account hardware or time constraints in applications, such as an IoT network equipped with cheap computation units or in a rapidly changing environment.…”

Decentralized Fictitious Play with Voluntary Communication in Random Communication Networks

“…Such metrics that are based on second order beliefs (estimating the estimates of the receiving agents) can provide similar benefits to communication efficiency in other decentralized game-theoretic learning algorithms based on, e.g., gradient descent [21]- [24], best-response [25], ADMM [26], and other adaptive strategies [27]. Indeed, communication-censoring based protocols that rely on some form of novelty of information metrics recently proved viable in reducing communication attempts in distributed stochastic gradient descent [28], [29] and ADMM [30] in the context of optimization. In the class of information exchange protocols considered here, while the novelty of information metric is sender specific, the metric on potential effect of information on other's assessment is receiving agent specific.…”