AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices

Abstract

In this work, we investigate the challenging problem of on-demand federated learning (FL) over heterogeneous edge devices with diverse resource constraints. We propose a cost-adjustable FL framework, named AnycostFL, that enables diverse edge devices to efficiently perform local updates under a wide range of efficiency constraints. To this end, we design the model shrinking to support local model training with elastic computation cost, and the gradient compression to allow parameter transmission with dynamic communication overhead. An enhanced parameter aggregation is conducted in an element-wise manner to improve the model performance. Focusing on Any-costFL, we further propose an optimization design to minimize the global training loss with personalized latency and energy constraints. By revealing the theoretical insights of the convergence analysis, personalized training strategies are deduced for different devices to match their locally available resources. Experiment results indicate that, when compared to the state-of-theart efficient FL algorithms, our learning framework can reduce up to 1.9 times of the training latency and energy consumption for realizing a reasonable global testing accuracy. Moreover, the results also demonstrate that, our approach significantly improves the converged global accuracy.

Adaptive Model Pruning for Communication and Computation Efficient Wireless Federated Learning

“…Moreover, the unstructured pruning approach usually results in irregular weight matrixes in the pruned models that are difficult to compress, which requires specialized hardware and software libraries to accelerate the training speed [24]. To effectively decrease computation and communication overhead, the structured model pruning approach [20]- [22], [25], [26] was developed to prune both filters in convolution layers and neurons in FC layers to generate sub-models for devices to train. Note that in centralized learning, pruning filters in convolution layers have been demonstrated can effectively accelerate the learning speed without sacrificing too much accuracy [24], [27].…”

“…The static model pruning approach in [21], [25] or local model composition approach in [28] distributed heterogeneous sub-models to devices for training and then aggregated them into a global inference model, which effectively reduced resource consumption for FL. The model shrinking and gradient compression approach in [26] enabled the local model training with elastic computation and communication overheads. The model pruning method in [22] dynamically adjusted the model size for resource-limited devices and significantly improved the cost-efficiency of FL.…”

An edge‐assisted federated contrastive learning method with local intrinsic dimensionality in noisy label environment

“…Since then, significant strides have been made to improve FL's performance. There have been many studies trying to improve the performance of FL from different perspectives, such as: model heterogeneity, 47,48 non independently and identically distributed (non-IID) data, 18,49 communication efficiency, [50][51][52] robust FL. 26,41 However, most of the existing studies in FL assume that every client has a clean dataset are not designed for tackling with noisy labels.…”