BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

Ying, Bicheng; Yuan, Kun; Hu, Hanbin; Chen, Yiming; Yin, Wotao

doi:10.48550/arxiv.2111.04287

Cited by 5 publications

(3 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, each node samples b = 16 data per iteration. Since this experiment is on a real distributed GPU system, we utilize BlueFog [Ying et al, 2021b] for the implementation of decentralized methods including topology organization, weight matrix generation, and efficient decentralized communication. For the implementation of C-SGD, we utilize PyTorch's native Distributed Data Parallel (DDP) module.…”

Section: Further Details Of Resnet-20 Experimentsmentioning

confidence: 99%

Heavy-Tail Phenomenon in Decentralized SGD

Gürbüzbalaban¹,

Hu²,

Şimşekli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recent theoretical studies have shown that heavy-tails can emerge in stochastic optimization due to 'multiplicative noise', even under surprisingly simple settings, such as linear regression with Gaussian data. While these studies have uncovered several interesting phenomena, they consider conventional stochastic optimization problems, which exclude decentralized settings that naturally arise in modern machine learning applications. In this paper, we study the emergence of heavy-tails in decentralized stochastic gradient descent (DE-SGD), and investigate the effect of decentralization on the tail behavior. We first show that, when the loss function at each computational node is twice continuously differentiable and strongly convex outside a compact region, the law of the DE-SGD iterates converges to a distribution with polynomially decaying (heavy) tails. To have a more explicit control on the tail exponent, we then consider the case where the loss at each node is a quadratic, and show that the tail-index can be estimated as a function of the step-size, batch-size, and the topological properties of the network of the computational nodes. Then, we provide theoretical and empirical results * The authors are in alphabetical order.

show abstract

Section: Further Details Of Resnet-20 Experimentsmentioning

confidence: 99%

Heavy-Tail Phenomenon in Decentralized SGD

Gürbüzbalaban¹,

Hu²,

Şimşekli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Nowadays, there are many FL frameworks. The most prominent TensorFlow Federated (TFF) [1], [2] and BlueFog [3] work well in cloud-edge continuum. However, they are not deployable to edge only, they are not supported on OS Windows, and they have numerous dependencies that make their installation far from trivial.…”

Section: Introductionmentioning

confidence: 99%

Developing Elementary Federated Learning Algorithms Leveraging the ChatGPT

Popovic,

Kastelan

et al. 2023

2023 31st Telecommunications Forum (TELFOR)

View full text Add to dashboard Cite

The Python Testbed for Federated Learning Algorithms is a simple Python FL framework easy to use by ML&AI developers who do not need to be professional programmers, and this paper shows that it is also amenable to emerging AI tools. In this paper, we successfully developed three elementary FL algorithms using the following three steps process: (i) specify context, (ii) ask ChatGPT to complete server and clients' callback functions, and (iii) verify the generated code.

show abstract

“…Decentralized optimization is an emerging learning paradigm in which each node only communicates with its immediate neighbors per iteration. By avoiding the central server and maintaining a more balanced communication between each pair of connected nodes, decentralized approaches can significantly speedup the training process of large-scale machine learning models [4,10,43]. Although decentralized optimization has been extensively studied in literature, its performance limits with time-varying communication patterns has not been fully explored.…”

Section: Introductionmentioning

confidence: 99%

Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks

Huang¹,

Yuan²

2022

Preprint

View full text Add to dashboard Cite

Decentralized optimization with time-varying networks is an emerging paradigm in machine learning. It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are moving. Federated learning can also be regarded as decentralized optimization with time-varying communication patterns alternating between global averaging and local updates.While numerous studies exist to clarify its theoretical limits and develop efficient algorithms, it remains unclear what the optimal complexity is for non-convex decentralized stochastic optimization over time-varying networks. The main difficulties lie in how to gauge the effectiveness when transmitting messages between two nodes via time-varying communications, and how to establish the lower bound when the network size is fixed (which is a prerequisite in stochastic optimization). This paper resolves these challenges and establish the first lower bound complexity. We also develop a new decentralized algorithm to nearly attain the lower bound, showing the tightness of the lower bound and the optimality of our algorithm.

show abstract

BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

Cited by 5 publications

References 65 publications

Heavy-Tail Phenomenon in Decentralized SGD

Heavy-Tail Phenomenon in Decentralized SGD

Developing Elementary Federated Learning Algorithms Leveraging the ChatGPT

Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks

Contact Info

Product

Resources

About