Asynchronous Federated Learning with Differential Privacy for Edge Intelligence

Li, Yanan; Yang, Shusen; Ren, Xuebin; Zhao, Cong

doi:10.48550/arxiv.1912.07902

Cited by 6 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [39], Mitliagkas et al demonstrated that the stale gradients could have the same effect as momentum when the staleness follows a geometric distribution. In fact, consistent stale gradients can boost the convergence, analogous to the function of a large momentum [39], [28]. However, when the model is close to the optimal solution, the momentum may fluctuate the training process.…”

Section: Interplay Of Non-iid Data and Stalenessmentioning

confidence: 99%

“…Since stale model is possibly farther from optimal solution and its derivative has a larger norm, stale gradients generally have larger norm for convex loss function. Therefore, the selected consistent stale gradients can help converge faster [39], [28].…”

Section: Basic Idea Of Wkaflmentioning

confidence: 99%

“…Besides the above synchronous methods, the K asynchronous FL (K-async FL) has also been extensively studied [25], [26], [27], [28], in which the server updates the global model once receiving the gradients from the first K clients and those clients who fail to participate in current iteration can continue their training for reducing the runtime in the next iteration. K-async FL can not only alleviate stragglers, but also save the total training time when the iteration time of clients follows a new-longer-than-used distribution [23].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Zhou,

Li,

Ren

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is an emerging privacy-preserving paradigm that enables multiple participants collaboratively to train a global model without uploading raw data. Considering heterogeneous computing and communication capabilities of different participants, asynchronous FL can avoid the stragglers effect in synchronous FL and adapts to scenarios with vast participants. Both staleness and non-IID data in asynchronous FL would reduce the model utility. However, there exists an inherent contradiction between the solutions to the two problems. That is, mitigating the staleness requires to select less but consistent gradients while coping with non-IID data demands more comprehensive gradients. To address the dilemma, this paper proposes a two-stage weighted K asynchronous FL with adaptive learning rate (WKAFL). By selecting consistent gradients and adjusting learning rate adaptively, WKAFL utilizes stale gradients and mitigates the impact of non-IID data, which can achieve multifaceted enhancement in training speed, prediction accuracy and training stability. We also present the convergence analysis for WKAFL under the assumption of unbounded staleness to understand the impact of staleness and non-IID data. Experiments implemented on both benchmark and synthetic FL datasets show that WKAFL has better overall performance compared to existing algorithms.

show abstract

Section: Interplay Of Non-iid Data and Stalenessmentioning

confidence: 99%

Section: Basic Idea Of Wkaflmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Zhou,

Li,

Ren

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The experiment is carried out on three real-world datasets, with the results demonstrating the high accuracy and efficiency of the scheme. Similarly, in [47], the convergence of AFL while adopting differential privacy is analyzed. Based on the analysis, a multi-stage adjustable private algorithm that dynamically changes the noise size and the learning rate is proposed to optimize the trade-off between model utility and privacy.…”

Section: Differential Privacy On Heterogeneous Devicesmentioning

confidence: 99%

Asynchronous Federated Learning on Heterogeneous Devices: A Survey

Xiang

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is experiencing a fast booming with the wave of distributed machine learning and ever-increasing privacy concerns. In the FL paradigm, global model aggregation is handled by a centralized aggregate server based on local updated gradients trained on local nodes, which mitigates privacy leakage caused by the collection of sensitive information. With the increased computing and communicating capabilities of edge and IoT devices, applying FL on heterogeneous devices to train machine learning models becomes a trend. The synchronous aggregation strategy in the classic FL paradigm cannot effectively use the resources, especially on heterogeneous devices, due to its waiting for straggler devices before aggregation in each training round. Furthermore, in real-world scenarios, the disparity of data dispersed on devices (i.e. data heterogeneity) downgrades the accuracy of models. As a result, many asynchronous FL (AFL) paradigms are presented in various application scenarios to improve efficiency, performance, privacy, and security. This survey comprehensively analyzes and summarizes existing variants of AFL according to a novel classification mechanism, including device heterogeneity, data heterogeneity, privacy and security on heterogeneous devices, and applications on heterogeneous devices. Finally, this survey reveals rising challenges and presents potentially promising research directions in this under-investigated field. CCS Concepts: • Computer systems organization → Distributed architectures; • Computing methodologies → Learning paradigms; • Networks → Mobile networks.

show abstract

“…They perform the convergence boosting by updates verification and weighted aggregation without any theoretical analysis. Li et al [17] aims to secure asynchronous edge-cloud collaborative federated learning with differential privacy. But they choose centralized learning and conduct analysis under the convex condition.…”

Section: Differentially Private Distributed Learningmentioning

confidence: 99%

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Xu¹,

Zhang²,

Wang³

2020

Preprint

View full text Add to dashboard Cite

As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. A popular distributed learning strategy is federated learning, where there is a central server storing the global model and a set of local computing nodes updating the model parameters with their corresponding data. The updated model parameters will be processed and transmitted to the central server, which leads to heavy communication costs. Recently, asynchronous decentralized distributed learning has been proposed and demonstrated to be a more efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD (ADPSGD) framework, or A(DP) 2 SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) 2 SGD also converges at the optimal O(1/ √ T ) rate as SGD. Empirically, A(DP) 2 SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments.

show abstract

Asynchronous Federated Learning with Differential Privacy for Edge Intelligence

Cited by 6 publications

References 35 publications

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data

Asynchronous Federated Learning on Heterogeneous Devices: A Survey

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Contact Info

Product

Resources

About