Exponential Graph is Provably Efficient for Decentralized Deep Training

Abstract

Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization. In this paper, we study so-called exponential graphs where every node is connected to O(log(n)) neighbors and n is the total number of nodes. This work proves such graphs can lead to both fast communication and effective averaging simultaneously. We also discover that a sequence of log(n) one-peer exponential graphs, in which each node communicates to one single neighbor per iteration, can together achieve exact averaging. This favorable property enables one-peer exponential graph to average as effective as its static counterpart but communicates more efficiently. We apply these exponential graphs in decentralized (momentum) SGD to obtain the state-of-the-art balance between per-iteration communication and iteration complexity among all commonly-used topologies. Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and highquality training. Our code is implemented through BlueFog and available at https://github.com/Bluefog-Lib/NeurIPS2021-Exponential-Graph.

BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

“…Push matrix W is also used with directed graph. 3) Standard weight matrix W satisfies both W 1 = 1 and 1 T W = 1 T and used for undirected graph, as well as special directed graphs such as the exponential graph [33]. See Fig.…”

“…When the network topology is sparse (e.g., a ring or a one-peer exponential graph [3], [33]), each partial averaging step (5) incurs O(1) latency and O(1) transmission time (the inverse of bandwidth), which are independent of n. Since each node only synchronizes with its direct neighbors, there is low synchronization overhead. effective in aggregating information than global averaging, some decentralized algorithms can match or exceed the performance of global-averaging-based distributed algorithms: [1], [29] established that decentralized SGD can achieve the same asymptotic linear speedup in convergence rate as (parameter server based) distributed SGD; [3], [33] used exponential graph topologies to realize both efficient communication and effective aggregation by partial averaging; [37], [38], [31], [39] improved the convergence rate of decentralized SGD by removing data heterogeneity between nodes; [40], [4], [30], [41] enhanced the effectiveness of partial averaging by periodically calling global averaging. BlueFog can implement all these algorithms including those use global averaging.…”

“…Prague [68] wraps all-reduce or broadcast operations, which is smart but relatively inefficient and restrictive. In particular, Prague forms a ring from a random subset of nodes and then ring-allreduces over them, thus ruling out other effective topologies such as the exponential graph [3], [33].…”

“…Code. We set the topology as the static exponential graph in the following DGD implementation, which is established in [33] to be both sparse and well-connected. Note that such static exponential graph and its associated weight matrix W has already been implemented in the BlueFog library.…”

“…It keeps evolving during the past two years, and the progress was reported in some conferences 1 . BlueFog has provided demos for several distributed and decentralized algorithms introduced in [32], and it has supported all deep learning experiments in [4], [5], [33]. All BlueFog code and supporting documents can be found at https://github.com/Bluefog-Lib.…”