FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning

Zhou, Yi; Ram, Parikshit; Salonidis, Theodoros; Baracaldo, Nathalie; Samulowitz, Horst; Ludwig, Heiko

doi:10.48550/arxiv.2112.08524

Cited by 2 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors in [22] disentangle the parameter set into local model parameters and global aggregation parameters, then update them iteratively with a communication-efficient algorithm. The authors in [25] introduce federated loss surface aggregation (FLoRA) as an FL-HPO solution framework. However, these works have not consider HPO rates or factors and how they can affect the performance in the FL systems.…”

Section: Related Workmentioning

confidence: 99%

Federated Hyperparameter Optimisation with Flower and Optuna

Parra-Ullauri

Zhang

Bravalheri

et al. 2023

Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing

View full text Add to dashboard Cite

Federated learning (FL) is an emerging distributed machine learning technique in which multiple clients collaborate to learn a model under the management of a central server. An FL system depends on a set of initial conditions (i.e., hyperparameters) that affect the system's performance. However, defining a good choice of hyperparameters for the central server and clients is a challenging problem. Hyperparameter tuning in FL often requires manual or automated searches to find optimal values. Nonetheless, a noticeable limitation is the high cost of algorithm evaluation for server and client models, making the tuning process computationally expensive and timeconsuming. We propose an implementation based on integrating the FL framework Flower, and the prime optimisation software Optuna for automated and efficient hyperparameter optimisation (HPO) in FL. Through this combination, it is possible to tune hyperparameters in both clients and server online, aiming to find the optimal values at runtime. We introduce the HPO factor to describe the number of rounds that the HPO will take place, and the HPO rate that defines the frequency for updating the hyperparameters and can be used for pruning. The HPO is managed by the FL server which updates clients' hyperparameters, with an HPO rate, using state-of-the-art optimisation algorithms enabled by Optuna. We tested our approach by updating multiple client models simultaneously in popular image recognition datasets which produced promising results compared to baselines.

show abstract

Section: Related Workmentioning

confidence: 99%

Federated Hyperparameter Optimisation with Flower and Optuna

Parra-Ullauri

Zhang

Bravalheri

et al. 2023

Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing

View full text Add to dashboard Cite

show abstract

“…As a result, most traditional black-box optimizers that require more than one full-fidelity trials are impractical in the FL setting [32]. Thus, multi-fidelity methods, particularly those capable of one-shot optimization [21,50], are more in demand in FedHPO.…”

Section: Uniqueness Of Federated Hyperparameter Optimizationmentioning

confidence: 99%

“…As mentioned in Section 3.1, tasks other than federated supervised learning will be incorporated. At the same time, we aim to extend FEDHPO-B to include different FL settings, e.g., HPO for vertical FL [50]. Another issue the current version has not touched on is the risk of privacy leakage caused by HPO methods [23], which we should provide related metrics and testbeds in the future.…”

mentioning

confidence: 99%

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

Wang¹,

Kuang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Hyperparameter optimization (HPO) is crucial for machine learning algorithms to achieve satisfactory performance, whose progress has been boosted by related benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for traditional centralized learning while ignoring federated learning (FL), a promising paradigm for collaboratively learning models from dispersed data. In this paper, we first identify some uniqueness of HPO for FL algorithms from various aspects. Due to this uniqueness, existing HPO benchmarks no longer satisfy the need to compare HPO methods in the FL setting. To facilitate the research of HPO in the FL setting, we propose and implement a benchmark suite FEDHPO-B that incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FEDHPO-B to benchmark a few HPO methods. We open-source FEDHPO-B at https://github.com/alibaba/FederatedScope/tree/master/benchmark/FedHPOB.

show abstract

FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning

Cited by 2 publications

References 25 publications

Federated Hyperparameter Optimisation with Flower and Optuna

Federated Hyperparameter Optimisation with Flower and Optuna

FedHPO-B: A Benchmark Suite for Federated Hyperparameter Optimization

Contact Info

Product

Resources

About