Secure Distributed Training at Scale

Abstract

Some of the hardest problems in deep learning can be solved with the combined effort of many independent parties, as is the case for volunteer computing and federated learning. These setups rely on high numbers of peers to provide computational resources or train on decentralized datasets. Unfortunately, participants in such systems are not always reliable. Any single participant can jeopardize the entire training run by sending incorrect updates, whether deliberately or by mistake. Training in presence of such peers requires specialized distributed training algorithms with Byzantine tolerance. These algorithms often sacrifice efficiency by introducing redundant communication or passing all updates through a trusted server. As a result, it can be infeasible to apply such algorithms to large-scale distributed deep learning, where models can have billions of parameters. In this work, we propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency. We rigorously analyze this protocol: in particular, we provide theoretical bounds for its resistance against Byzantine and Sybil attacks and show that it has a marginal communication overhead. To demonstrate its practical effectiveness, we conduct large-scale experiments on image classification and language modeling in presence of Byzantine attackers.

Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top

“…As. 2.4 Gorbunov et al [2021a] assume additionally that the tails of the noise distribution in stochastic gradients are sub-quadratic.…”

“…This approach is extended to the case of heterogeneous data and aggregators agnostic to the noise level by , and propose an extension to the decentralized optimization over fixed networks. Gorbunov et al [2021a] propose an alternative approach based on the usage of AllReduce [Patarasuk and Yuan, 2009] with additional verifications of correctness and show that their algorithm has complexity not worse than Parallel-SGD when the target accuracy is small enough. Wu et al [2020] are the first who applied variance reduction mechanism to tolerate Byzantine attacks (see the discussion above Q1).…”

“…Wu et al [2020] are the first who applied variance reduction mechanism to tolerate Byzantine attacks (see the discussion above Q1). We also refer reader to , Rajput et al, 2019, Rodríguez-Barroso et al, 2020, Xu and Lyu, 2020, Alistarh et al, 2018, Allen-Zhu et al, 2021, Regatti et al, 2020, Yang and Bajwa, 2019a,b, Gupta et al, 2021, Peng et al, 2021 for other advances in Byzantine-robustness (see the detailed summaries in , Gorbunov et al, 2021a). We further progress the field by obtaining new theoretical SOTA convergence results in our work.…”