Dynamic Resource Allocation for Hierarchical Federated Learning

Lim, Wei Yang Bryan; Ng, Jer Shyuan; Xiong, Zehui; Niyato, Dusit; Guo, Song; Leung, Cyril; Chen, Miao

doi:10.1109/msn50589.2020.00038

Cited by 86 publications

(180 citation statements)

References 11 publications

Supporting

Mentioning

179

Contrasting

Unclassified

Order By: Relevance

“…Significant attention has been paid towards developing federated optimization techniques. Such work has focused on various aspects, including communication-efficiency (Konečnỳ et al, 2016;McMahan et al, 2017;Basu et al, 2019;Laguel et al, 2021), data and systems heterogeneity (Li et al, 2020aKarimireddy et al, 2020b;Hsu et al, 2019;Karimireddy et al, 2020a;Li et al, 2020b;Li and Wang, 2019), and fairness (Li et al, 2020c;Hu et al, 2020). We provide a description of some relevant methods in Section 2, and defer readers to recent surveys such as (Kairouz et al, 2021) and (Li et al, 2020a) for additional background.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

On Large-Cohort Training for Federated Learning

Charles¹,

Garrett²,

Huo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods that effectively utilize larger cohort sizes? We give partial answers to these questions based on extensive empirical evaluation. Our work highlights a number of challenges stemming from the use of larger cohorts. While some of these (such as generalization issues and diminishing returns) are analogs of large-batch training challenges, others (including training failures and fairness concerns) are unique to federated learning.

show abstract

Section: Related Workmentioning

confidence: 99%

“…One critical issue in FL is fairness across clients, as minimizing (1) may disadvantage some clients (Mohri et al, 2019;Li et al, 2020c). Intuitively, large-cohort training methods may be better suited for ensuring fairness, since a greater fraction of the population is allowed to contribute to the model at each round.…”

Section: Fairness Concernsmentioning

confidence: 99%

On Large-Cohort Training for Federated Learning

Charles¹,

Garrett²,

Huo³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…doing federated NAS, and tuning initialization-based meta-learning algorithms such as Reptile and MAML. Lastly, any work on FL comes with privacy and fairness risks due its frequent use of sensitive data; thus any application of our work must consider tools being developed by the community for mitigating such issues [32,37].…”

Section: Discussionmentioning

confidence: 99%

Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing

Khodak¹,

Tu²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a novel connection to the neural architecture search technique of weight-sharing, we introduce a new method, FedEx, to accelerate federated hyperparameter tuning that is applicable to widely-used federated optimization methods such as FedAvg and recent variants. Theoretically, we show that a FedEx variant correctly tunes the on-device learning rate in the setting of online convex optimization across devices. Empirically, we show that FedEx can outperform natural baselines for federated hyperparameter tuning by several percentage points on the Shakespeare, FEMNIST, and CIFAR-10 benchmarks-obtaining higher accuracy using the same training budget.Preprint. Under review.

show abstract

“…First, our model connects to models incentivizing parity in test accuracy in federated learning. (Li et al, 2019) proposes a variant of a global objective criterion to induce parity in test accuracy between clients in heterogeneous settings. While this literature also studies the supply side, i.e., who receives models, the paper implicitly assumes that the preference for a high-quality model is homogeneous across clients and tries to induce parity without any monetary transfers.…”

Section: Literature Reviewmentioning

confidence: 99%

“…In contrast to several contributions on incentive design for federated learning (Chen et al, 2020;Jiao et al, 2019;Li et al, 2019), we assume that clients submit models trained on their entire datasetwith no need to incentivize them to do so. Under this assumption, we design a system to collect resources from clients.…”

Section: Introductionmentioning

confidence: 99%

Prior-Independent Auctions for the Demand Side of Federated Learning

Haupt¹,

Mugunthan²

2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a paradigm that allows distributed clients to learn a shared machine learning model without sharing their sensitive training data. While largely decentralized, FL requires resources to fund a central orchestrator or to reimburse contributors of datasets to incentivize participation. Inspired by insights from prior-free auction design, we propose a mechanism, FIPFA (Federated Incentive Payments via Prior-Free Auctions), to collect monetary contributions from self-interested clients. The mechanism operates in the semi-honest trust model and works even if clients have a heterogeneous interest in receiving high-quality models, and the server does not know the clients' level of interest. We run experiments on the MNIST dataset to test clients' model quality under FIPFA and FIPFA's incentive properties.

show abstract

Dynamic Resource Allocation for Hierarchical Federated Learning

Cited by 86 publications

References 11 publications

On Large-Cohort Training for Federated Learning

On Large-Cohort Training for Federated Learning

Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing

Prior-Independent Auctions for the Demand Side of Federated Learning

Contact Info

Product

Resources

About