Pufferfish: Communication-efficient Models At No Extra Cost

Wang, Hongyi; Agarwal, Saurabh; Papailiopoulos, Dimitris S.

doi:10.48550/arxiv.2103.03936

Cited by 3 publications

(3 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Communication-Efficient FL Algorithms -Many recent works proposed sparsification and quantization methods specifically designed for FL (Alistarh et al 2017(Alistarh et al , 2018Albasyoni et al 2020;Wangni et al 2017;Wang et al 2018;Wang, Agarwal, and Papailiopoulos 2021;Alistarh et al 2017;Wen et al 2017;Reisizadeh et al 2020). These methods are also called sketched approach (Konečnỳ et al 2016).…”

Section: Related Workmentioning

confidence: 99%

“…et al 2018), FedNova (Wang et al 2020), and SCAFFOLD (Karimireddy et al 2020), periodically average the full local solutions across all the clients. Many communicationefficient FL strategies, such as gradient (model) sparsification (Wangni et al 2017;Wang et al 2018;Alistarh et al 2018), low-rank approximation (Vogels, Karimireddy, and Jaggi 2020;Wang, Agarwal, and Papailiopoulos 2021), and quantization (Alistarh et al 2017;Wen et al 2017;Albasyoni et al 2020;Reisizadeh et al 2020) techniques, also periodically aggregate the compressed form of the full local solutions. Adaptive model aggregation techniques (Wang and Joshi 2018a;Haddadpour et al 2019) adjust the aggregation interval at run-time to reduce the total number of communications, however, they still aggregate the full local models at once.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning

Lee

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

In Federated Learning (FL), a common approach for aggregating local solutions across clients is periodic full model averaging. It is, however, known that different layers of neural networks can have a different degree of model discrepancy across the clients. The conventional full aggregation scheme does not consider such a difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parameters that are similar across the clients does not make meaningful training progress while increasing the communication cost. We propose FedLAMA, a layer-wise adaptive model aggregation scheme for scalable FL. FedLAMA adjusts the aggregation interval in a layer-wise manner, jointly considering the model discrepancy and the communication cost. This fine-grained aggregation strategy enables to reduce the communication cost without significantly harming the model accuracy. Our extensive empirical study shows that, as the aggregation interval increases, FedLAMA shows a remarkably smaller accuracy drop than the periodic full aggregation, while achieving comparable communication efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning

Lee

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…The reconstructed parameters are local to the clients, and never sent to the server. Wang et al [2021a] proposed training low-rank, pre-factorized deep networks to reduce communication in distributed learning. Other methods, like compression, and knowledge distillation have been used in FL to reduce the communication costs.…”

Section: Related Workmentioning

confidence: 99%

Efficient and Private Federated Learning with Partially Trainable Networks

Sidahmed¹,

Xu²,

Garg³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to 46× reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT can be particularly interesting for pushing the limitations of overparameterization in on-device learning.

show abstract

FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning

Nam¹,

Ye-Bin²,

2021

Preprint

View full text Add to dashboard Cite

To overcome the burdens on frequent model uploads and downloads during federated learning (FL), we propose a communication-efficient re-parameterization, FedPara. Our method re-parameterizes the model's layers using low-rank matrices or tensors followed by the Hadamard product. Different from the conventional lowrank parameterization, our method is not limited to low-rank constraints. Thereby, our FedPara has a larger capacity than the low-rank one, even with the same number of parameters. It can achieve comparable performance to the original models while requiring 2.8 to 10.1 times lower communication costs than the original models, which is not achievable by the traditional low-rank parameterization. Moreover, the efficiency can be further improved by combining our method and other efficient FL techniques because our method is compatible with others. We also extend our method to a personalized FL application, pFedPara, which separates parameters into global and local ones. We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.

show abstract

Pufferfish: Communication-efficient Models At No Extra Cost

Cited by 3 publications

References 64 publications

Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning

Layer-Wise Adaptive Model Aggregation for Scalable Federated Learning

Efficient and Private Federated Learning with Partially Trainable Networks

FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning

Contact Info

Product

Resources

About