Over-the-Air Federated Learning from Heterogeneous Data

Abstract

Federated learning (FL) is a framework for distributed learning of centralized models. In FL, a set of edge devices train a model using their local data, while repeatedly exchanging their trained updates with a central server. This procedure allows tuning a centralized model in a distributed fashion without having the users share their possibly private data. In this paper, we focus on over-the-air (OTA) FL, which has been suggested recently to reduce the communication overhead of FL due to the repeated transmissions of the model updates by a large number of users over the wireless channel. In OTA FL, all users simultaneously transmit their updates as analog signals over a multiple access channel, and the server receives a superposition of the analog transmitted signals. However, this approach results in the channel noise directly affecting the optimization procedure, which may degrade the accuracy of the trained model. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local stochastic gradient descent (SGD) FL algorithm, introducing precoding at the users and scaling at the server, which gradually mitigates the effect of the noise. We analyze the convergence of COTAF to the loss minimizing model and quantify the effect of a statistically heterogeneous setup, i.e. when the training data of each user obeys a different distribution. Our analysis reveals the ability of COTAF to achieve a convergence rate similar to that achievable over error-free channels. Our simulations demonstrate the improved convergence of COTAF over vanilla OTA local SGD for training using non-synthetic datasets. Furthermore, we numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL. I. INTRODUCTIONRecent years have witnessed unprecedented success of machine learning methods in a broad range of applications [2]. These systems utilize highly parameterized models, such as deep neural networks (DNNs), trained using massive data sets. In many applications, samples are available at remote users, e.g. smartphones, and the common strategy is to gather these samples at a computationally powerful server, where the model is trained [3]. Often, data sets contain private information, and thus the user may not be willing to share them with the server. Furthermore, sharing massive data sets can result in a substantial burden on the communication links between the users and the server. To allow centralized training without data sharing, federated learning (FL) was proposed in [4] as a method combining A short version of this paper that introduces the algorithm for i.i.d. data and preliminary simulation results was accepted for presentation in the 2020 IEEE Global Communications Conference (GLOBECOM) [1].

Bayesian Over-The-Air Computation

“…If we define the first and second sample moments of the symbols transmitted by the m-th device as in (11), V can be written in the form of (10).…”

“…Prior works on the aligned OAC [1], [2], [4], [6], [8], [10], [11], [13], [14], [22], [23] or the misaligned OAC [9] rely exclusively on the maximum likelihood (ML) estimator to recover the arithmetic sum of the transmitted signals from different devices. ML estimation, however, is much susceptible to noise.…”

Accelerated Gradient Descent Learning over Multiple Access Fading Channels

“…where E N is a transmission energy-coefficient set to satisfy the energy requirement, e −jφ n,k is used to correct the phase reflection to yield coherent aggregated signals at the receiver, as suggested in past studies (e.g., [8], [16], [33], [41], [52], [53]), and s(t) = [s 1 (t), ..., s d (t)] T is a column vector of d orthogonal baseband equivalent normalized waveforms, as suggested in [8]. Also, the fact that DRAFT AGMA does not use power control or beamforming to correct the channel gain as in [3]- [5], [7],…”

Distributed Learning over a Wireless Network with FSK-Based Majority Vote

“…To overcome the impact of the multipath channel on the transmitted signals, the symbols on the OFDM subcarriers are multiplied with the inverse of the channel coefficients and the subcarriers that fade are excluded from the transmissions, which is known as truncated-channel inversion (TCI) in the literature. In [8], an additional time-varying precoder is applied along with TCI to facilitate the aggregation. In [9], it is proposed to sparsify the gradient estimates and project the resultant sparse vector into a low-dimensional vector to reduce the bandwidth.…”