Communication-Efficient Randomized Algorithm for Multi-Kernel Online Federated Learning

Abstract

Online federated learning (OFL) is a promising framework to learn a sequence of global functions using distributed sequential data at local devices. In this framework, we first introduce a single kernel-based OFL (termed S-KOFL) by incorporating the random-feature (RF) approximation, online gradient descent (OGD), and federated averaging (FedAvg) properly. However, it is nontrivial to develop a communication-efficient method with multiple kernels. One can construct a multi-kernel method (termed vM-KOFL) by following the extension principle in the centralized counterpart. This vanilla method is not practical as the communication overhead grows linearly with the size of a kernel dictionary. Moreover, this problem is not addressed via the existing communication-efficient techniques in federated learning such as quantization or sparsification. Our major contribution is to propose a novel randomized algorithm (named eM-KOFL), which can enjoy the advantage of multiple kernels while having a similar communication overhead with S-KOFL. It is theoretically proved that eM-KOFL yields the same asymptotic performance as vM-KOFL, i.e., both methods achieve an optimal sublinear regret bound. Mimicking the key principle of eM-KOFL efficiently, pM-KOFL is presented. Via numerical tests with real datasets, we demonstrate that pM-KOFL can yield the same performances as vM-KOFL and eM-KOFL on various online learning tasks while having the same communication overhead as S-KOFL. These suggest the practicality of the proposed pM-KOFL.

Addressing modern and practical challenges in machine learning: A survey of online federated and transfer learning

“…Statistical Heterogeneity [101,125,127,128,129,130] System Heterogeneity [121,127,130,131,10,132,133] Privacy Guarantees [134,135] mapping h should be carefully selected in accordance with the model parameter structures [132], from which each device k can estimate the label y t+1 k of a newly arrived data x k+1 t in real-time. One of the most commonly used mappings in the standard FL system is FedAvg [34], which averages the aggregated local parameter sets:…”

“…Air quality datasets collected from weather sensors in different countries were used in [127] and [132] to predict the level of pollutants in the air.…”

“…Given the achievements of OTL in domain shift scenarios and OFL in data privacy protection, it is reasonable to apply these methods to industrial engineering tasks, and Table 4 summarizes the detailed sub-scenarios of OTL and OFL applications in industrial engineering. OFL has been used in a variety of data-sensitive industry domains, including environmental protection [127,132], and unmanned aerial vehicle (UAV) control [148,125]. OTL applications, on the other hand, frequently appeared in industrial situations involving domain shift problems, such as sentiment analysis [110,102,9,107,116].…”

Tighter Regret Analysis and Optimization of Online Federated Learning

“…For instance, online learning is necessary when data is generated as a function of time (e.g., time-series predictions) [19]- [21] and when large-scale data makes it hard to carry out data analytic in batch form [22]. Recently, this challenging problem has been actively investigated using random-feature based kernel learning for various network architectures such as centralized, fully decentralized, and federated learning [23]- [26]. In centralized network, online multiple kernel learning (OMKL) (a.k.a.…”

“…Clearly, due to the advantage of using multiple kernels, DOMKL significantly outperforms RFF-DOKL. Especially in [26], online federated learning (OFL), which is closely related to the subject of this paper, was investigated, wherein a communication-efficient randomized algorithm (named PM-KOFL) was proposed. This algorithm achieves an optimal performance asymptotically while having much lower communication overhead than a vanilla method.…”