Model Pruning Enables Efficient Federated Learning on Edge Devices

Jiang, Yuang; Wang, Shiqiang; Valls, Víctor; Ko, Bong Jun; Lee, Wei-Han; Leung, Kin K.; Tassiulas, Leandros

doi:10.1109/tnnls.2022.3166101

Cited by 181 publications

(69 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are also some works focusing on practical aspects such as model compression and sparsification (Han, Wang, and Leung 2020;Jiang and Agrawal 2018;Jiang et al 2020;Konecny et al 2016) and partial worker participation (Bonawitz et al 2019;Chen et al 2020). These algorithms and techniques are orthogonal to our work and may be applied together with H-SGD.…”

Section: Related Workmentioning

confidence: 98%

Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

Wang

Wang²,

Chen

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Hierarchical SGD (H-SGD) has emerged as a new distributed SGD algorithm for multi-level communication networks. In H-SGD, before each global aggregation, workers send their updated local models to local servers for aggregations. Despite recent research efforts, the effect of local aggregation on global convergence still lacks theoretical understanding. In this work, we first introduce a new notion of "upward" and "downward" divergences. We then use it to conduct a novel analysis to obtain a worst-case convergence upper bound for two-level H-SGD with non-IID data, non-convex objective function, and stochastic gradient. By extending this result to the case with random grouping, we observe that this convergence upper bound of H-SGD is between the upper bounds of two single-level local SGD settings, with the number of local iterations equal to the local and global update periods in H-SGD, respectively. We refer to this as the "sandwich behavior". Furthermore, we extend our analytical approach based on "upward" and "downward" divergences to study the convergence for the general case of H-SGD with more than two levels, where the "sandwich behavior" still holds. Our theoretical results provide key insights of why local aggregation can be beneficial in improving the convergence of H-SGD.

show abstract

Section: Related Workmentioning

confidence: 98%

Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

Wang

Wang²,

Chen

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [27], the authors provide an optimization problem whose goal is to minimize the total energy consumption of the system under a latency constraint; to solve the problem, an iterative algorithm is proposed where, at every step, closed-form solutions for time allocation, bandwidth allocation, power control, computation frequency, and learning accuracy are derived. The work in [28] The work in [30] proposes a FL approach with adaptive and distributed parameter pruning, which adapts the model size during FL to reduce both communication and computation overhead and minimize the overall training time, while maintaining a similar accuracy as the original model. In [31], the authors consider two transmission protocols for edge devices to upload model parameters to edge server, based on non orthogonal multiple access and time division multiple access, respectively.…”

Section: Related Workmentioning

confidence: 99%

Lyapunov-Based Optimization of Edge Resources for Energy-Efficient Adaptive Federated Learning

Battiloro

Lorenzo

Merluzzi

et al. 2023

IEEE Trans. on Green Commun. Netw.

View full text Add to dashboard Cite

The aim of this paper is to propose a novel dynamic resource allocation strategy for energyefficient adaptive federated learning at the wireless network edge, with latency and learning performance guarantees. We consider a set of devices collecting local data and uploading processed information to an edge server, which runs stochastic gradient-based algorithms to perform continuous learning and adaptation. Hinging on Lyapunov stochastic optimization tools, we dynamically optimize radio parameters (e.g., set of transmitting devices, transmit powers, bits, and rates) and computation resources (e.g., CPU cycles at devices and at server) in order to strike the best trade-off between power, latency, and performance of the federated learning task. The framework admits both a model-based implementation, where the learning performance metrics are available in closed-form, and a data-driven approach, which works with online estimates of the learning performance of interest. The method is then customized to the case of federated least mean squares (LMS) estimation, and federated training of deep convolutional neural networks. Numerical results illustrate the effectiveness of our strategy to perform energy-efficient, low-latency, adaptive federated learning at the wireless network edge.

show abstract

“…In order to reduce the communication load during the propagation phase, many researches have applied sparsification and model pruning into FL. Authors in [19] inherit the simple regularization method to develop pruneFL algorithm. This algorithm can reduce the number of parameters up to 7 times.…”

Section: Introductionmentioning

confidence: 99%

HCFL: A High Compression Approach for Communication-Efficient Federated Learning in Very Large Scale IoT Networks

Nguyen¹,

Lee²,

Pham³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a new artificial intelligence concept that enables Internet-of-Things (IoT) devices to learn a collaborative model without sending the raw data to centralized nodes for processing. Despite numerous advantages, low computing resources at IoT devices and high communication costs for exchanging model parameters make applications of FL in massive IoT networks very limited. In this work, we develop a novel compression scheme for FL, called high-compression federated learning (HCFL), for very large scale IoT networks. HCFL can reduce the data load for FL processes without changing their structure and hyperparameters. In this way, we not only can significantly reduce communication costs, but also make intensive learning processes more adaptable on low-computing resource IoT devices. Furthermore, we investigate a relationship between the number of IoT devices and the convergence level of the FL model and thereby better assess the quality of the FL process. We demonstrate our HCFL scheme in both simulations and mathematical analyses. Our proposed theoretical research can be used as a minimum level of satisfaction, proving that the FL process can achieve good performance when a determined configuration is met. Therefore, we show that HCFL is applicable in any FL-integrated networks with numerous IoT devices.

show abstract

Model Pruning Enables Efficient Federated Learning on Edge Devices

Cited by 181 publications

References 24 publications

Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGD

Lyapunov-Based Optimization of Edge Resources for Energy-Efficient Adaptive Federated Learning

HCFL: A High Compression Approach for Communication-Efficient Federated Learning in Very Large Scale IoT Networks

Contact Info

Product

Resources

About