Decentralized Non-Convex Learning with Linearly Coupled Constraints

Abstract

Motivated by the need for decentralized learning, this paper aims at designing a distributed algorithm for solving nonconvex problems with general linear constraints over a multi-agent network. In the considered problem, each agent owns some local information and a local variable for jointly minimizing a cost function, but local variables are coupled by linear constraints. Most of the existing methods for such problems are only applicable for convex problems or problems with specific linear constraints. There still lacks a distributed algorithm for such problems with general linear constraints and under nonconvex setting. In this paper, to tackle this problem, we propose a new algorithm, called "proximal dual consensus" (PDC) algorithm, which combines a proximal technique and a dual consensus method. We build the theoretical convergence conditions and show that the proposed PDC algorithm can converge to an -Karush-Kuhn-Tucker solution within O(1/ ) iterations. For computation reduction, the PDC algorithm can choose to perform cheap gradient descent per iteration while preserving the same order of O(1/ ) iteration complexity. Numerical results are presented to demonstrate the good performance of the proposed algorithms for solving a regression problem and a classification problem over a network where agents have only partial observations of data features. * Jiawei Zhang and Songyang Ge contribute equally. Part of the work was presented in IEEE ICASSP 2020 [1].

Gradient and Variable Tracking with Multiple Local SGD for Decentralized Non-Convex Learning

“…Besides, extension of the LSGT algorithm for the learning problem over the hybrid data, i.e., the MUST algorithm, is investigated in Sec. 5. Numerical results are given in Sec.…”

“…The proposed LSGT algorithm is presented in Algorithm 1. Comparing to the vanilla stochastic GT method in (5), in the proposed LSGT algorithm, each agent n executes E consecutive steps of SGD within each communication round. In particular, in each step q ∈ [E], agent n performs gradient descent along the direction v r,q−1 n as in (7a).…”

“…This phenomenon makes it difficult to store all the data in one device and process it by one processor. In view of this, distributed optimization methods arouse significant attention in various signal processing and machine learning fields, see, e.g., [1][2][3][4][5][6][7]. In particular, by considering a multi-agent network with N agents linked via a connected graph, most of the existing distributed optimization methods focus on solving the following problem min y∈R p F (y) GT [11] ns-cvx.…”