FedML: A Research Library and Benchmark for Federated Machine Learning

He, Chaoyang; Li, Songze; So, Jinhyun; Zeng, Xiao; Zhang, Mi; Wang, Hongyi; Wang, Xiaoyang; Vepakomma, Praneeth; Singh, Abhishek; Qiu, Hang; Zhu, Xinghua; Wang, Jianzong; Shen, Li; Zhao, Peilin; Kang, Yan; Liu, Yang; Raskar, Ramesh; Yang, Qiang; Annavaram, Murali; Avestimehr, Salman

doi:10.48550/arxiv.2007.13518

Cited by 107 publications

(160 citation statements)

References 108 publications

Supporting

Mentioning

159

Contrasting

Order By: Relevance

“…In this section, we conduct extensive experiments to verify the efficacy of the proposed FedSpa. Our implementation of FedSpa is based on an open-source FL simulator FedML [27]. We fix the dense ratio of FedSpa (DST), FedSpa (RSM), and the final dense ratio of Fed-SubAvg both to 0.5 (i.e., 50% of parameters are pruned) in our main evaluation.…”

Section: Methodsmentioning

confidence: 99%

Achieving Personalized Federated Learning with Sparse Local Models

Huang¹,

Liu²,

Shen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge. Instead of training an intact (or dense) PFL model, FedSpa only maintains a fixed number of active parameters throughout training (aka sparse-to-sparse training), which enables users' models to achieve personalization with cheap communication, computation, and memory cost. We theoretically show that the iterates obtained by FedSpa converge to the local minimizer of the formulated SPFL problem at the rate of O( 1 √ T). Comprehensive experiments demonstrate that FedSpa significantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Achieving Personalized Federated Learning with Sparse Local Models

Huang¹,

Liu²,

Shen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…All our experiments are based on a non-IID data distribution among FL clients. We have used latent Dirichlet Distribution (LDA), which is a common data distribution in FL to generate non-IID data across clients [10], [39].…”

Section: Methodsmentioning

confidence: 99%

SPIDER: Searching Personalized Neural Architecture for Federated Learning

Mushtaq¹,

He²,

Ding³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, in this work, we introduce SPIDER, an algorithmic framework that aims to Search PersonalIzed neural architecture for feDERated learning. SPIDER is designed based on two unique features:(1) alternately optimizing one architecture-homogeneous global model (Supernet) in a generic FL manner and one architecture-heterogeneous local model that is connected to the global model by weight sharing-based regularization (2) achieving architecture-heterogeneous local model by a novel neural architecture search (NAS) method that can select optimal subnet progressively using operation-level perturbation on the accuracy value as the criterion. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods, and the searched personalized architectures are more inference efficient.

show abstract

“…As federated learning has raised lots of attention over the recent years, researchers also start to investigate building efficient libraries (He et al, 2020;Beutel et al, 2020) and systems (Bonawitz et al, 2019b;a;Hiessl et al, 2020). Several seminal work has also been done on designing better communication protocol (Konečnỳ et al, 2016), optimization algorithms (Li et al, 2018), and improving model robustness (Konstantinidis & Ramamoorthy, 2021).…”

Section: Related Workmentioning

confidence: 99%

An Efficient and Robust System for Vertically Federated Random Forest

Yao¹,

Wang²,

Dai³

et al. 2022

Preprint

View full text Add to dashboard Cite

As there is a growing interest in utilizing data across multiple resources to build better machine learning models, many vertically federated learning algorithms have been proposed to preserve the data privacy of the participating organizations. However, the efficiency of existing vertically federated learning algorithms remains to be a big problem, especially when applied to large-scale real-world datasets. In this paper, we present a fast, accurate, scalable and yet robust system for vertically federated random forest. With extensive optimization, we achieved 5× and 83× speed up over the SOTA SecureBoost model (Cheng et al., 2019) for training and serving tasks. Moreover, the proposed system can achieve similar accuracy but with favorable scalability and partition tolerance. Our code has been made public to facilitate the development of the community and the protection of user data privacy.

show abstract

FedML: A Research Library and Benchmark for Federated Machine Learning

Cited by 107 publications

References 108 publications

Achieving Personalized Federated Learning with Sparse Local Models

Achieving Personalized Federated Learning with Sparse Local Models

SPIDER: Searching Personalized Neural Architecture for Federated Learning

An Efficient and Robust System for Vertically Federated Random Forest

Contact Info

Product

Resources

About