FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Abstract

The underlying assumption of recent federated learning (FL) paradigms is that local models usually share the same network architecture as the global model, which becomes impractical for mobile and IoT devices with different setups of hardware and infrastructure. A scalable federated learning framework should address heterogeneous clients equipped with different computation and communication capabilities. To this end, this paper proposes FEDHM, a novel federated model compression framework that distributes the heterogeneous low-rank models to clients and then aggregates them into a global full-rank model. Our solution enables the training of heterogeneous local models with varying computational complexities and aggregates a single global model. Furthermore, FEDHM not only reduces the computational complexity of the device, but also reduces the communication cost by using low-rank models. Extensive experimental results demonstrate that our proposed FEDHM outperforms the current pruning-based FL approaches in terms of test Top-1 accuracy (4.6% accuracy gain on average), with smaller model size (1.5× smaller on average) under various heterogeneous FL settings.

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

“…The major difference to its use for inference is that here, the low-rank NN is updated during training, and the low-rank updates are applied to the full model on the server. Yao et al [101] present FedHM, where they create low-complexity submodels on the server by doing a low-rank factorization of the full model. Layer parameters with dimensions 𝑚 × 𝑛 are decomposed into two matrices with dimensions 𝑚 × 𝑟 and 𝑟 × 𝑛.…”

“…The attributes scale and granularity are often neglected, are hidden behind the technique, and lack discussion in the papers. The reported scale in the resources supported by the techniques ranges from 4× − 25× [12,41,52,61,71,77,79,80,85,101] up to 100× − 250× [25,87], yet it remains unclear whether training at such high scales is still effective. Hence, while all approaches show the effectiveness of their solution in certain scenarios, it often remains unclear whether devices with low resources or stale devices can make a meaningful contribution that advances the global model.…”

Memory-adaptive Depth-wise Heterogenous Federated Learning

“…Such edge devices are typically resource-constrained, e.g., the computing, communication, and memory capacities are limited. Several research efforts have been conducted to enhance the computation and communication efficiency of cross-device FL via model updates sparsification, quantization, and low-rank factorization [25,51,56,21]. Training deep neural networks requires high memory consumption [47].…”

FedorAS: Federated Architecture Search under system heterogeneity

“…As such, lately, there has been a line of work focusing on this very problem, where the discrepancy between the dynamics of different clients affects the convergence rate or fairness of the deployed system. Specifically, such solutions draw from efficient ML and attempt to dynamically alter the footprint of local models my means of structured (PruneFL [37]), unstructured (Adaptive Federated Dropout [13]) or importance-based pruning (FjORD [33]), quantisation (AQFL [2]), low-rank factorisation (FedHM [76]), sparsity-inducing training (ZeroFL [63]) or distillation (GKT [31]). However, each approach has limitations, either because they involve extra training overhead [31] and residence of multiple DNN copies in memory [2], or because they require specialised hardware for performance gains ( [63,13]).…”

“…However, each approach has limitations, either because they involve extra training overhead [31] and residence of multiple DNN copies in memory [2], or because they require specialised hardware for performance gains ( [63,13]). Last, some of the architectural changes proposed may not offer the degrees of freedom that NAS exposes [33,76].…”