Distributed Optimization Over Time-Varying Graphs With Imperfect Sharing of Information

Reisizadeh, Hadi; Touri, Behrouz; Mohajer, Soheil

doi:10.1109/tac.2022.3207866

Cited by 7 publications

(8 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This assumption is specific to the imperfect information sharing setup and is considered recently in [26], [29], [30].…”

Section: Assumptionsmentioning

confidence: 99%

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Chellapandi¹,

Upadhyay²,

Hashemi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Decentralized learning and optimization is a central problem in control that encompasses several existing and emerging applications, such as federated learning. While there exists a vast literature on this topic and most methods centered around the celebrated average-consensus paradigm, less attention has been devoted to scenarios where the communication between the agents may be imperfect. To this end, this paper presents three different algorithms of Decentralized Federated Learning (DFL) in the presence of imperfect information sharing modeled as noisy communication channels. The first algorithm, Federated Noisy Decentralized Learning (FedNDL1), comes from the literature, where the noise is added to their parameters to simulate the scenario of the presence of noisy communication channels. This algorithm shares parameters to form a consensus with the clients based on a communication graph topology through a noisy communication channel. The proposed second algorithm ( FedNDL2) is similar to the first algorithm but with added noise to the parameters, and it performs the gossip averaging before the gradient optimization. The proposed third algorithm (FedNDL3), on the other hand, shares the gradients through noisy communication channels instead of the parameters. Theoretical and experimental results demonstrate that under imperfect information sharing, the third scheme that mixes gradients is more robust in the presence of a noisy channel compared with the algorithms from the literature that mix the parameters.

show abstract

“…This assumption is specific to the imperfect information sharing setup and is considered recently in [26], [29], [30].…”

Section: Assumptionsmentioning

confidence: 99%

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Chellapandi¹,

Upadhyay²,

Hashemi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Based on this, the convergence rate of the algorithm is given in Theorem 4, and the related results are not provided for distributed stochastic optimization even when no privacy protection is considered. Note that the convergence rate for distributed optimization with non-vanishing noises is studied in [29], [32], where σ k " Oppk `a2 q η q, η " 0. Then, the convergence rate studied in this paper is nontrivial and more general than the one in [29], [32].…”

Section: B Convergence Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Distributed (stochastic) optimization has been widely used in various fields, such as big data analytics, finance, and distributed learning [25]- [32]. At present, there are many important techniques to solve distributed stochastic optimization, such as stochastic approximation [29]- [32] and time-varying sample-size. As a standard variance reduction technique, timevarying sample-size schemes have gained increasing research interests and have been widely used to solve various problems, such as large-scale machine learning [33], stochastic optimization [34]- [37], and stochastic generalized equations [38].…”

Section: Introductionmentioning

confidence: 99%

“…Compared with the existing distributed stochastic optimization algorithms with or without privacy protection [27], [39], [40], we present the mean-square convergence rate of the algorithm. Furthermore, compared with [29], [32], we give the convergence rate with more general noises. The remaining sections of this paper are organized as follows: Section II introduces the problem formulation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Privacy-Preserving Distributed Online Stochastic Optimization With Time-Varying Distributions

Wang

Liu

Han

et al. 2023

IEEE Trans. Control Netw. Syst.

View full text Add to dashboard Cite

Differentially private distributed stochastic optimization has become a hot topic due to the urgent need of privacy protection in distributed stochastic optimization. In this paper, two-time scale stochastic approximation-type algorithms for differentially private distributed stochastic optimization with time-varying sample sizes are proposed using gradient-and output-perturbation methods. For both gradient-and output-perturbation cases, the convergence of the algorithm and differential privacy with a finite cumulative privacy budget ε for an infinite number of iterations are simultaneously established, which is substantially different from the existing works. By a time-varying sample sizes method, the privacy level is enhanced, and differential privacy with a finite cumulative privacy budget ε for an infinite number of iterations is established. By properly choosing a Lyapunov function, the algorithm achieves almostsure and mean-square convergence even when the added privacy noises have an increasing variance. Furthermore, we rigorously provide the mean-square convergence rates of the algorithm and show how the added privacy noise affects the convergence rate of the algorithm. Finally, numerical examples including distributed training on a benchmark machine learning dataset are presented to demonstrate the efficiency and advantages of the algorithms.

show abstract

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Chellapandi

Upadhyay

Hashemi

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

A novel Decentralized Noisy Model Update Tracking Federated Learning algorithm (FedNMUT) is proposed that is tailored to function efficiently in the presence of noisy communication channels that reflect imperfect information exchange. This algorithm uses gradient tracking to minimize the impact of data heterogeneity while minimizing communication overhead. The proposed algorithm incorporates noise into its parameters to mimic the conditions of noisy communication channels, thereby enabling consensus among clients through a communication graph topology in such challenging environments. FedNMUT prioritizes parameter sharing and noise incorporation to increase the resilience of decentralized learning systems against noisy communications. Theoretical results for the smooth non-convex objective function are provided by us, and it is shown that the ϵ−stationary solution is achieved by our algorithm at the rate of O 1 √ T , where T is the total number of communication rounds. Additionally, via empirical validation, we demonstrated that the performance of FedNMUT is superior to the existing state-of-the-art methods and conventional parameter-mixing approaches in dealing with imperfect information sharing. This proves the capability of the proposed algorithm to counteract the negative effects of communication noise in a decentralized learning framework.Imperfect information exchange, such as noisy or quantized communication, has been examined in the context of average consensus algorithms within distributed frameworks. Yet, the ramifications of varying noise levels remain underexplored. Moreover, existing research, primarily focused on consensus issues, does not fully address the complex challenges encountered in contemporary decentralized optimization and learning paradigms [23], [24]. In contrast to Federated Learning (FL), where server assistance is common, Decentralized Federated Learning (DFL) operates without a central server, with each client acting autonomously, processing local Stochastic Gradient Descent (SGD) or its variants on its data and interacting directly with neighboring clients.In our previous paper [25], we performed a comparative study of three proposed algorithms for DFL under imperfect communication conditions, typified by noisy channels. These algorithms-FedNDL1, FedNDL2, and FedNDL3-differ in their handling of noise and parameter sharing, demonstrating varying degrees of resilience to communication noise. In this paper, we propose a novel algorithm that employs the Gradient Tracking method in DFL and compare its performance against the previously mentioned algorithms. C. Paper's ContributionsThis paper introduces a novel algorithm that employs the Gradient Tracking method in DFL, considering the impact of communication noise. Previous studies have evaluated the effectiveness of two-time scale methods in DFL with noisy channels. However, these investigations were limited by inflexible assumptions such as strong convexity in papers such as [26]- [29]. These assumptions are rarely satisfied in practi...

show abstract

Distributed Optimization Over Time-Varying Graphs With Imperfect Sharing of Information

Cited by 7 publications

References 22 publications

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Privacy-Preserving Distributed Online Stochastic Optimization With Time-Varying Distributions

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

Contact Info

Product

Resources

About