Exploring Parameter-Efficient Fine-tuning for Improving Communication Efficiency in Federated Learning

Abstract

Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. However, this can quickly put a massive communication burden on the system, especially if more capable models beyond very small MLPs are employed. Recently, the use of pre-trained models has been shown effective in federated learning optimization and improving convergence. This opens the door for new research questions. Can we adjust the weight-sharing paradigm in federated learning, leveraging strong and readilyavailable pre-trained models, to significantly reduce the communication burden while simultaneously achieving excellent performance? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning. Specifically, we systemically evaluate the performance of several parameter-efficient fine-tuning methods across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

“…Parameter-Efficient Finetuning (PEFT) for Federated Learning. PEFT has been well studied in centralized machine learning (Houlsby et al 2019;Liu et al 2022;Sung, Cho, and Bansal 2022), while its application on FL remains under-explored. Most of the prior work rudimentarily adapted PEFT for FL and focused on single-modal tasks:…”

“…(1) Image classification. Sun et al 2022) evaluate the existing PEFT baselines combined with FL, while (Guo et al 2022;Guo, Guo, and Wang 2023;Li et al 2023;Lu et al 2023) finetune the CLIP model (Radford et al 2021) via tuning and communicating only small amount of learnable (personalized) prompts. (Su et al 2022) addresses the problem of heterogeneous client images by injecting lightweight adaptation modules (adapters) (Houlsby et al 2019).…”

FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models

“…With this trait, PETuning methods can be utilized to mitigate the communication overhead in FL, which primarily relies on the size of model update parameters. In the field of computer vision, Sun et al (2022) present the FedPEFT framework by injecting three PETuning methods (i.e., Bais, Adapter, Prompt) of the visual pre-trained models into FL, and find that lightweight PETuning methods in FL can significantly reduce the communication burden while maintaining performance and performing better in different FL settings. Meanwhile, Chen et al (2022c) extend PETuning methods to the visual language model in FL and show that PETuning can facilitate a fast convergence rate.…”

Language-Guided Transformer for Federated Multi-Label Classification

“…To mitigate performance degradation induced by data heterogeneity (e.g., label distribution skew) on FL, various works (Li et al 2020;Karimireddy et al 2020;Gao et al 2022;Li, He, and Song 2021;Huang, Ye, and Du 2022;Su et al 2022;Mendieta et al 2022;Wang et al 2020;Tan et al 2021;Zhuang et al 2021;Luo et al 2021) have been proposed to address this impact. For example, FedProx (Li et al 2020) introduces a regularization term to mitigate the label distribution skew issues from the local learning step of FL.…”

“…• To the best of our knowledge, we are the first to tackle the problem of label discrepancy across different clients for multi-label FL. (Su et al 2022;Guo et al 2023;Chen et al 2022;Sun et al 2022) utilize vision-language pre-trained models, which perform prompt tuning during training rounds and thus reduce the number of learnable parameters. FedAPT (Su et al 2022) learns a global adaptive network with the global prompt under the FL setting, then the framework generates domainspecific prompts for CLIP to handle the domain shift issue under FL.…”