Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networks

Abstract

This paper aims to integrate two synergetic technologies, federated learning (FL) and width-adjustable slimmable neural network (SNN) architectures. FL preserves data privacy by exchanging the locally trained models of mobile devices. By adopting SNNs as local models, FL can flexibly cope with the time-varying energy capacities of mobile devices. Combining FL and SNNs is however non-trivial, particularly under wireless connections with time-varying channel conditions. Furthermore, existing multi-width SNN training algorithms are sensitive to the data distributions across devices, so are ill-suited to FL. Motivated by this, we propose a communication and energy efficient SNN-based FL (named SlimFL) that jointly utilizes superposition coding (SC) for global model aggregation and superposition training (ST) for updating local models. By applying SC, SlimFL exchanges the superposition of multiple width configurations that are decoded as many as possible for a given communication throughput. Leveraging ST, SlimFL aligns the forward propagation of different width configurations, while avoiding the interwidth interference during back propagation. We formally prove the convergence of SlimFL. The result reveals that SlimFL is not only communication-efficient but also can counteract non-IID data distributions and poor channel conditions, which is also corroborated by simulations.

AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices

“…Besides, FedMask proposes to train a personalized mask for each device to improve the test accuracy on the local dataset [31]. Recently, model structure pruning enables multiple devices with different model architectures to train a shared global model [32], [33]. Such methods can reduce the cost of local training, but how to customize optimal training strategies (e.g., gradient compression and model pruning policy) for different learning scenarios is still unknown.…”

FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

“…However, the architectures of local and global models are still restricted by the same model architecture. SlimFL [30] integrates slimmable neural network (SNN) architectures [31] into FL, adapting the widths of local neural networks based on resource limitations. In [32], FjORD leverages Ordered Dropout and a self-distillation method to determine the model widths.…”

Parallelized and randomized adversarial imitation learning for safety-critical self-driving vehicles

“…One of the major previous research results on self-driving and autonomous driving is rule-based driving policy optimization. However, the policy has serious problems for coping with timevarying environments, i.e., extremely large observation spaces and action spaces which can introduce high computational complexity for training [11]- [16]. Recently, deep reinforcement learning (DRL) based algorithms have been proposed that utilize powerful function approximations such as neural networks, allow the vehicular supervisor to train robust driving policies [17]- [23].…”