On Biased Compression for Distributed Learning

Abstract

In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning. However, despite the fact biased compressors often show superior performance in practice when compared to the much more studied and understood unbiased compressors, very little is known about them. In this work we study three classes of biased compression operators, two of which are new, and their performance when applied to (stochastic) gradient descent and distributed (stochastic) gradient descent. We show for the first time that biased compressors can lead to linear convergence rates both in the single node and distributed settings. Our distributed SGD method enjoys the ergodic rate Kµ, where δ is a compression parameter which grows when more compression is applied, L and µ are the smoothness and strong convexity constants, C captures stochastic gradient noise (C = 0 if full gradients are computed on each node) and D captures the variance of the gradients at the optimum (D = 0 for over-parameterized models). Further, via a theoretical study of several synthetic and empirical distributions of communicated gradients, we shed light on why and by how much biased compressors outperform their unbiased variants. Finally, we propose a new highly performing biased compressor-combination of Top-k and natural dithering-which in our experiments outperforms all other compression techniques.

Communication Compression for Distributed Nonconvex Optimization

“…However, such a simple strategy does not converge to the accurate solution due to the compression error, and even may leads to divergence as the compression error would accumulate. Examples have been provided in [11], [12] to illustrate this. Therefore, communication compression in decentralized algorithms has gained considerable attention recently.…”

“…[33]- [37], and also includes biased and non-contractive compressors, such as the norm-sign compressor. Moreover, it is straightforward to check that the class of compressors satisfying Assumption 1 also covers the three classes of biased compressors considered in [12]. In other words, Assumption 1 is weaker than various commonly used assumptions for compressors in the literature.…”

Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

“…To remove the synchronization barrier, asynchronous update methods are proposed [52,64,68,76,120,123]. There are also approaches that combines multiple strategies listed above [15,17,44,56,80]. On the other hand, researches about model parallelism attempt to study how to allocate model parameters and training computation across compute units in a cluster to maximize training throughput and minimize communication overheads.…”

Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization

“…The common choice of C(•) can be top-k (Basu et al, 2019) or sign operation (leads to signSGD (Bernstein et al, 2018)). Although this naive compression method is intuitive, it can diverge in practice, even in simple quadratic problems (Beznosikov et al, 2020) or constraint linear problems (Karimireddy et al, 2019). Intuitively speaking, one of the major drawbacks of naive compression is that the compression error is accumulating during the training process.…”

“…It directly compresses the local fresh gradient on each worker before uploading the gradient to the server. However, the compression would slow down the convergence or even diverge (Beznosikov et al, 2020) due to the loss of information at each compression step. Later on, error feedback strategy (Stich et al, 2018;Karimireddy et al, 2019) was proposed to alleviate this problem and reduce the information loss by proposing a compensating error sequence.…”