Achieving Personalized Federated Learning with Sparse Local Models

Abstract

Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge. Instead of training an intact (or dense) PFL model, FedSpa only maintains a fixed number of active parameters throughout training (aka sparse-to-sparse training), which enables users' models to achieve personalization with cheap communication, computation, and memory cost. We theoretically show that the iterates obtained by FedSpa converge to the local minimizer of the formulated SPFL problem at the rate of O( 1 √ T). Comprehensive experiments demonstrate that FedSpa significantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.

Efficient Federated Learning for distributed NeuroImaging Data

“…Moreover, in Li et al (2020a), each client trains a personalized mask to maximize the performance only on the local data. A few recent works Bibikar et al (2022); Huang et al (2022); Qiu et al (2022); Li et al (2020a) also attempted to leverage sparse training within the FL setting as well. In particular Li et al (2020a) implemented randomly initialized sparse mask, FedDST Bibikar et al (2022) built on the idea of RigL Evci et al (2020) and mostly focussed on magnitude pruning on the server-side resulting in similar constraints and Ohib et al (2023) uses sparse gradients to efficiently train in a federated learning setting.…”

FedABC: Targeting Fair Competition in Personalized Federated Learning

“…To deal with heterogeneity in FL, numerous solutions have been proposed, such as the ones that constrain the direction of local model update to align the local and global optimization objectives (Li et al 2020;Karimireddy et al 2020;Acar et al 2021;Zhang et al 2022;Liu et al 2022). Personalized federated learning (PFL) (Smith et al 2017;Huang et al 2022b;Dai et al 2022) is a promising solution to addresses this challenge by jointly learning multiple personalized models (PMs), one for each client. For instance, references (Collins et al 2021;Liang et al 2020;Sun et al 2021)…”