Networked Federated Learning

Abstract

Many important application domains generate distributed collections of heterogeneous local datasets. These local datasets are often related via an intrinsic network structure that arises from domain-specific notions of similarity between local datasets. Different notions of similarity are induced by spatio-temporal proximity, statistical dependencies or functional relations. We use this network structure to adaptively pool similar local datasets into nearly homogenous training sets for learning tailored models. Our main conceptual contribution is to formulate networked federated learning using the concept of generalized total variation (GTV) minimization as a regularizer. This formulation is highly flexible and can be combined with almost any parametric model including Lasso or deep neural networks. We unify and considerably extend some well-known approaches to federated multi-task learning. Our main algorithmic contribution is a novel federated learning algorithm which is well suited for distributed computing environments such as edge computing over wireless networks. This algorithm is robust against model misspecification and numerical errors arising from limited computational resources including processing time or wireless channel bandwidth. As our main technical contribution, we offer precise conditions on the local models as well on their network structure such that our algorithm learns nearly optimal local models. Our analysis reveals an interesting interplay between the (information-) geometry of local models and the (cluster-) geometry of their network.Preprint. Under review.

A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

“…Besides that, there are also several other works studying FMTL. [29] proposed a framework for generalized total variation minimization, which is useful in FMTL networks. [30] introduced a FMTL algorithm to deal with the issues of accuracy, fairness and robustness in FL.…”

“…On the other hand, problem (1) is a generalization of the problem in [39] where several algorithms are developed for strongly convex objectives. Problem (1) is also similar to the generalized total variation minimization problem [29] which is solved by a primal-dual method for convex objectives.…”

“…We first show benefits of FedU in FMTL setting by comparing FedU with Local model (training one separate model per client), Global model (training one single model on centralized data), and MOCHA, the conventional FMTL algorithm [16]. Note that the performance results of the FMTL algorithm in [29] and MOCHA are reported similar. We evaluate FedU on a wide range of values of η ∈ {5.10 −3 , 10 −3 , 5.10 −2 , 10 −2 , 10 −1 , 1} and compare with others using their best fine-tuned parameters.…”