From distributed machine learning to federated learning: a survey

Liu, Ji; Huang, Jizhou; Zhou, Yang; Li, Xuhong; Ji, Shilei; Xiong, Haoyi; Dou, Dejing

doi:10.1007/s10115-022-01664-x

Cited by 129 publications

(61 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proof. We first perform an inequality transformation on F (w k+1 ) − F (w k ) and then convert the equation to contain only the similar items of ∇F (w k ) 2 . For details, see Appendix A…”

Section: A Convergence Analysismentioning

confidence: 99%

“…F EDERATED learning is a paradigm of distributed machine learning that allows the decentralized clients to train a global model using their own local data, thus alleviating the problems of data silos and user privacy [1], [2]. In FL, the clients' availability and decentralization bring great communication overhead to the network, and it can be reduced by increasing the local computing [3].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective

Luo¹,

Cheng²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) is a distributed machine learning framework to alleviate the data silos, where decentralized clients collaboratively learn a global model without sharing their private data. However, the clients' Non-Independent and Identically Distributed (Non-IID) data negatively affect the trained model, and clients with different numbers of local updates may cause significant gaps to the local gradients in each communication round. In this paper, we propose a Federated Vectorized Averaging (FedVeca) method to address the above problem on Non-IID data. Specifically, we set a novel objective for the global model which is related to the local gradients. The local gradient is defined as a bi-directional vector with step size and direction, where the step size is the number of local updates and the direction is divided into positive and negative according to our definition. In FedVeca, the direction is influenced by the step size, thus we average the bi-directional vectors to reduce the effect of different step sizes. Then, we theoretically analyze the relationship between the step sizes and the global objective, and obtain upper bounds on the step sizes per communication round. Based on the upper bounds, we design an algorithm for the server and the client to adaptively adjusts the step sizes that make the objective close to the optimum. Finally, we conduct experiments on different datasets, models and scenarios by building a prototype system, and the experimental results demonstrate the effectiveness and efficiency of the FedVeca method.

show abstract

Section: A Convergence Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective

Luo¹,

Cheng²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To that end, the concept of federated learning can be utilized. This concept is a branch of machine learning that deals with training a model across multiple decentralized edge devices (or servers) without exchanging local data samples between them (Yang et al, 2019;Zhang et al, 2021;Wahab et al, 2021;Liu et al, 2022). Although such privacy preserving decentralized and collaborative machine learning applications become more of a main stream approach in advertising, financial and many other industries with personalized dominance, it is relatively uncharted in advancing functionalities and providing technical support to forge a cross-domain, cross-data and cross-enterprise digital twin ecosystem.…”

Section: Introductionmentioning

confidence: 99%

Decentralized digital twins of complex dynamical systems

San¹,

Pawar²,

Rasheed³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we introduce a decentralized digital twin (DDT) framework for dynamical systems and discuss the prospects of the DDT modeling paradigm in computational science and engineering applications. The DDT approach is built on a federated learning concept, a branch of machine learning that encourages knowledge sharing without sharing the actual data. This approach enables clients to collaboratively learn an aggregated model while keeping all the training data on each client. We demonstrate the feasibility of the DDT framework with various dynamical systems, which are often considered prototypes for modeling complex transport phenomena in spatiotemporally extended systems. Our results indicate that federated machine learning might be a key enabler for designing highly accurate decentralized digital twins in complex nonlinear spatiotemporal systems.

show abstract

“…It supports efficient machine learning among multiple participants or computing nodes under the premise of the legality and compliance [1]. Recently, the FL algorithms optimization and privacy-preserving methods have achieved rich results in theoretical research and practical applications [2][3][4][5][6]. Federated learning is expected to become the basis of the next generation of collaborative artificial intelligence algorithms and networks.…”

Section: Introductionmentioning

confidence: 99%

Restrictively Self-sampled and Compressed Local Differential Privacy in Federated Learning

Zhao

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

As a popular machine learning framework, federated learning (FL) enables clients to conduct cooperative training without sharing data, thus having higher security than conventional machine learning. However, by sharing parameters in the federated learning process, the attacker can still obtain private information from the sensitive data of participants by reverse parsing. Recently, local differential privacy (LDP) has worked well in preserving privacy for federated learning. However, it faces the inherent problem of balancing privacy, model performance, and algorithm efficiency. In this paper, we propose a novel local differential privacy method in federated learning (SLDP-FL), which achieves the privacy amplification effect by the client self-sampling and provides compressed and private parameters in each iteration by a compressed LDP mechanism. Thereby, it improves the model performance as well as efficiency observably.Moreover, we theoretically analyze the relationship between the model accuracy and client self-sampling probability. A restrictive client self-sampling technology is proposed, which eliminates the randomness of self-sampling probability settings in existing studies and improves the utilization of the federated system. Comprehensive experiments on MNIST and Fashion-MNIST datasets show that the SLDP-FL optimizes the existing federated learning framework through compression mechanism and self-sampling technique with restrictive probability since it is superior to the current algorithms' accuracy and convergence and communication efficiency.

show abstract

From distributed machine learning to federated learning: a survey

Cited by 129 publications

References 95 publications

FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective

FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective

Decentralized digital twins of complex dynamical systems

Restrictively Self-sampled and Compressed Local Differential Privacy in Federated Learning

Contact Info

Product

Resources

About