Joint Optimization of Communications and Federated Learning Over the Air

Abstract

Federated learning (FL) is an attractive paradigm for making use of rich distributed data while protecting data privacy. Nonetheless, nonideal communication links and limited transmission resources have become the bottleneck of the implementation of fast and accurate FL. In this paper, we study joint optimization of communications and FL based on analog aggregation transmission in realistic wireless networks. We first derive a closed-form expression for the expected convergence rate of FL over the air, which theoretically quantifies the impact of analog aggregation on FL. Based on the analytical result, we develop a joint optimization model for accurate FL implementation, which allows a parameter server to select a subset of workers and determine an appropriate power scaling factor. Since the practical setting of FL over the air encounters unobservable parameters, we reformulate the joint optimization of worker selection and power allocation using controlled approximation. Finally, we efficiently solve the resulting mixed-integer programming problem via a simple yet optimal finite-set search method by reducing the search space. Simulation results show that the proposed solutions developed for realistic wireless analog channels outperform a benchmark method, and achieve comparable performance of the ideal case where FL is implemented over noise-free wireless channels.

Distributed Learning in Wireless Networks: Recent Progress and Future Challenges

“…In [13], the authors employ random projection of the sparsified model updates at the devices, which allows the devices to significantly reduce the bandwidth requirement without sacrificing the performance. The authors in [109] first analyzed how user selction and transmit power affect the convergence of AirComp based FL and then optimized these wireless factors to improve the performance of AirComp based FL. The work in [110] studied the use of 1-bit compressive sensing (CS) for analog ML model aggregation thus reduce the size of FL parameters transmitted over wireless links.…”

Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications

“…The convergence behavior demonstrates that the noisy iterates typically introduce non-negligible optimality gap in various FL algorithms, e.g., vanilla gradient method [174], quantized gradient method [175], sparsified gradient method [173], and operator splitting method [87]. The optimality gap can be further controlled by transmit power allocation [176], [41], [173], model aggregation receiver beamforming design [177], [31], [178], and device scheduling [178], [31], [179]. Besides, channel perturbation in algorithm iterates can also serve as the mechanism to design saddle points escaping algorithms [94], thereby establishing global optimality for training the non-convex over-parameterized neural networks in high-dimensional statistical settings [180].…”

“…Specifically, for edge training systems via AirComp, the global model aggregation errors due to the wireless channel fading and noise will cause learning performance degradation [45], [87]. The optimality gap (i.e., the distance between the current iterate and the desired solution), characterized by the convergence behavior of the global iterate, can be further controlled by various resource allocation schemes, including edge devices transmit power control [41], [237], edge server receive beamforming [31], [179], passive beamforming at RIS [48], [178], as well as device scheduling policy [31], [48]. For digital design of the edge training system, the optimality basically depends on the edge devices selection, packet errors in the uplink transmission, and model parameter partition, for which user scheduling [238], power control [88], batchsize selection [239], aggregation frequency control [240], and bandwidth allocation [241] were provided to improve the accuracy in the edge training process.…”

BEV-SGD: Best Effort Voting SGD for Analog Aggregation Based Federated Learning against Byzantine Attackers

“…Benefitting from communication-efficient gradient aggregation, FLOA as a cross-disciplinary topic has attracted growing research interests in the fields of communications, optimization, and machine learning, such as power control [16], [19], [29], devices scheduling [16], [18], [25],…”

“…For instance, a broadband analog aggregation scheme for power control and device scheduling in FLOA is proposed in [18], where a set of tradeoffs between communications and learning are discussed. In [16], convergence analysis quantifies the impact of AirComp on FL and then a joint optimization of communication and learning for the optimal power scaling and device scheduling is proposed. Considering energy-constrained local devices, an energy-aware device scheduling strategy is proposed in [25] to maximize the average number of workers scheduled for gradient update.…”