PyFed: extending PySyft with N-IID Federated Learning Benchmark

Bouraqqadi, Houda; Berrag, Ayoub; Mhaouach, Mohamed; Bouhoute, Afaf; Fardousse, Khalid; Berrada, Ismaïl

doi:10.21428/594757db.9c5550b5

“…To the best of our knowledge, Motley is the first benchmark that aims to develop baselines for personalized FL in cross-device and cross-silo federated settings. Previous FL benchmarks (see, e.g, [7,8,9,11,24,25,29,38,40] and the references therein) do not contain personalization baselines and focus on the standard FedAvg [50] algorithm and its variations [43,47,55]. Motley is designed with a focus on ease-of-use and reproducibility.…”

Section: Introductionmentioning

confidence: 99%

Motley: Benchmarking Heterogeneity and Personalization in Federated Learning

Wu¹,

Li²,

Charles³

et al. 2022

Preprint

View full text Add to dashboard Cite

Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning.

show abstract

“…This is mainly due to the following four reasons: 1) Tools not open-source, like Sherpa-ai [63] 2) Missing repositories: is the case of tools that have not yet released their codes after the paper publication: Chiron [64], FedHealth [65], FAE [66], GENO [67], FedTGan [68], IPLS [70] and FLPytorch [71]. 3) Coherent but not suitable: is the case of LEAF [45], FL-Bench [69], and PyFed [57] which are positioned for benchmarking purposes and therefore, might lack of essential features for conducting more extensive research activities. FedGraphNN [46] is a sub-project of the bigger initiative called FedML [40] already included in this survey.…”

Section: Easy Integration With Other Toolsmentioning

confidence: 99%

FeLebrities: a user-centric assessment of Federated Learning frameworks

Riviera¹,

Menegaz²,

Galazzo³

2023

Preprint

0

View full text Add to dashboard Cite

<p>Federated Learning (FL) is a new paradigm that aims at solving the data access problem. It is gaining an increasing interest in a variety of research fields, including the Biomedical and Financial environments, where lots of valuable data sources are available but not often directly accessible due to the regulations that protect sensitive information. FL provides a wayout enabling the processing and sharing of data modeling solutions moving the focus from data to models. The FL paradigm involves different entities (institutions) holding proprietary datasets, contributing with each other to locally train a copy of a shared Artificial Intelligence (AI) model. Although there are different studies in the literature that suggest how to conceptually implement and orchestrate a federation, fewer efforts have been made on practical implications. With the ambition of helping accelerating the exploitation of FL frameworks, this paper proposes a survey of public tools that are currently available, an objective ranking based on current state of user preferences and the assessment of the growth trend of the tool popularity over a six months time window. Finally, a ranking of the tools maturity is derived based on key aspects to consider when building a FL pipeline.</p>

show abstract

“…This is mainly due to the following four reasons: 1) Tools not open-source, like Sherpa-ai [63] 2) Missing repositories: is the case of tools that have not yet released their codes after the paper publication: Chiron [65], FedHealth [66], FAE [67], GENO [68], FedTGan [69] and IPLS [71]. 3) Coherent but not suitable: is the case of LEAF [45], FL-Bench [70], and PyFed [57] which are positioned for benchmarking purposes and therefore, might lack of essential features for conducting more extensive research activities. FedGraphNN [46] is a sub-project of the bigger initiative called FedML [40] already included in this survey.…”

Section: Easy Integration With Other Toolsmentioning

confidence: 99%

FeLebrities: a user-centric assessment of Federated Learning frameworks

Riviera¹,

Menegaz²,

Galazzo³

2022

Preprint

0

View full text Add to dashboard Cite

<p>Federated Learning (FL) is a new paradigm that aims at solving the data access problem. It is gaining an increasing interest in a variety of research fields, including the Biomedical and Financial environments, where lots of valuable data sources are available but not often directly accessible due to the regulations that protect sensitive information. FL provides a wayout enabling the processing and sharing of data modeling solutions moving the focus from data to models. The FL paradigm involves different entities (institutions) holding proprietary datasets, contributing with each other to locally train a copy of a shared Artificial Intelligence (AI) model. Although there are different studies in the literature that suggest how to conceptually implement and orchestrate a federation, fewer efforts have been made on practical implications. With the ambition of helping accelerating the exploitation of FL frameworks, this paper proposes a survey of public tools that are currently available, an objective ranking based on current state of user preferences and the assessment of the growth trend of the tool popularity over a six months time window. Finally, a ranking of the tools maturity is derived based on key aspects to consider when building a FL pipeline.</p>

show abstract

PyFed: extending PySyft with N-IID Federated Learning Benchmark

Cited by 6 publications

References 2 publications

Motley: Benchmarking Heterogeneity and Personalization in Federated Learning

Motley: Benchmarking Heterogeneity and Personalization in Federated Learning

FeLebrities: a user-centric assessment of Federated Learning frameworks

FeLebrities: a user-centric assessment of Federated Learning frameworks

Contact Info

Product

Resources

About