ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity

Abstract

When the available hardware cannot meet the memory and compute requirements to efficiently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed. In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional servergrade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm. While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training. Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side. In this work, we present the first study on the unique aspects that arise when introducing sparsity at training time in FL workloads. We then propose ZeroFL, a framework that relies on highly sparse operations to accelerate on-device training. Models trained with ZeroFL and 95% sparsity achieve up to 2.3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.

Efficient Federated Learning for distributed NeuroImaging Data

“…Moreover, in Li et al (2020a), each client trains a personalized mask to maximize the performance only on the local data. A few recent works Bibikar et al (2022); Huang et al (2022); Qiu et al (2022); Li et al (2020a) also attempted to leverage sparse training within the FL setting as well. In particular Li et al (2020a) implemented randomly initialized sparse mask, FedDST Bibikar et al (2022) built on the idea of RigL Evci et al (2020) and mostly focussed on magnitude pruning on the server-side resulting in similar constraints and Ohib et al (2023) uses sparse gradients to efficiently train in a federated learning setting.…”

Limitations and Future Aspects of Communication Costs in Federated Learning: A Survey

“…Tensor pruning involves setting entire blocks of the model to zero, which can be useful when the model has a structured block-wise pattern. Structured pruning can preserve the underlying structure of the model and can result in higher compression rates than random pruning [195]. Still, it may require more complex implementation and may introduce more errors in the model or gradient values.…”