A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

Dinh, Canh T.; Vu, Tung T.; Tran, Nguyen H.; Dao, Minh N.; Zhang, Hongyu

doi:10.1109/tnnls.2022.3224252

Cited by 38 publications

(39 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A numerical comparison of test accuracy over five independent runs is shown in Table I. FLACC is compared to NoFed, FedAvg, two state-of-the-art personalized FL methods pFedMe [17], Per-FedAvg [20], and and three state-of-the-art clustered FL methods IFCA [24], FL+HC [21], and CFL [26]. For NoFed, each client learns only from its local data, and model averaging is not performed.…”

Section: Resultsmentioning

confidence: 99%

“…The second line of research is pluralistic and learns individually personalized models for each client. This approach is termed as personalized federated learning and can be achieved via multi-task learning [16], model regularization [17], contextualization [18], local fine tuning [19], and meta learning [20]. The third line of research is clustered FL, which assumes that groups of clients share one data distribution.…”

Section: Related Workmentioning

confidence: 99%

“…( 14) and Eq. (17). For s estimated underlying clusters in the federation, it is recommended to have at least |M (t) | = s clients participating in each round, with |M (t) | ≥ 2s to ensure ideal qualitative clustering.…”

Section: Practical Considerationsmentioning

confidence: 99%

See 2 more Smart Citations

A Greedy Agglomerative Framework for Clustered Federated Learning

Mehta

Shao

2023

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Federated learning (FL) has received widespread attention for decentralized training of deep learning models across devices while preserving privacy. Industrial big data in key applications like healthcare, smart manufacturing, autonomous driving, and robotics is inherently multi-source and heterogeneous. Recent studies have shown that the quality of the global FL model deteriorates in the presence of such non-IID data. To address this, we present a novel clustered FL framework called Federated Learning via Agglomerative Client Clustering (FLACC). FLACC greedily agglomerates clients or groups of clients based on their gradient updates while learning the global FL model. Once the clustering is complete, each cluster is separated into its own federation, allowing clients with similar underlying distributions to train together. In contrast with existing methods, FLACC does not require the number of clusters to be specified a priori, can handle client fractions, and is robust to hyperparameter tuning. We demonstrate the efficacy of this framework through extensive experiments on three benchmark FL datasets and an FL case study simulated using industrial mixed fault classification data. Qualitative clustering results show that FLACC accurately identifies clusters in the presence of various statistical heterogeneities in the client data. Quantitative results show that FLACC outperforms vanilla FL and state-of-the-art personalized and clustered FL methods, even when the underlying clustering structure is not apparent.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Greedy Agglomerative Framework for Clustered Federated Learning

Mehta

Shao

2023

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

“…Full Model Personalization usually requires each client to train a personalized model and a global model, where the global model is used to prevent the personalized model from overfitting to its local data. It includes methods based on meta learning (Fallah et al, 2020), model mixture Deng et al, 2020;Mansour et al, 2020), global reguarlization (Li et al, 2021a), mean regularization (T Dinh et al, 2020; and clustering (Sattler et al, 2020;Ghosh et al, 2020). However, these methods induce high costs by training two full models in each client and communicating the full model.…”

Section: Related Workmentioning

confidence: 99%

“…Existing works in pFL commonly use full model personalization, where each client trains a personalized model as well as a copy of the global model from the server for regularization (Li et al, 2021a;T Dinh et al, 2020). However, these methods are parameter-expensive, leading to high computational and communicational costs, which is impractical for clients with limited computation resources and network bandwidth (Kairouz et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

Xie¹,

Huang²,

Wenda³

et al. 2023

Preprint

View full text Add to dashboard Cite

Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high communication and computation costs or (2) overfit to local data, which can be limited in scope, and are vulnerable to evolved test samples with natural shifts. In this paper, we propose PERADA, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts. PERADA reduces the costs by leveraging the power of pretrained models and only updates and communicates a small number of additional parameters from adapters. PERADA has good generalization since it regularizes each client's personalized adapter with a global adapter, while the global adapter uses knowledge distillation to aggregate generalized information from all clients. Theoretically, we provide generalization bounds to explain why PERADA improves generalization, and we prove its convergence to stationary points under non-convex settings. Empirically, PERADA demonstrates competitive personalized performance (+4.85% on CheXpert) and enables better out-of-distribution generalization (+5.23% on CIFAR-10-C) on different datasets across natural and medical domains compared with baselines, while only updating 12.6% of parameters per model based on the adapter.

show abstract

Stock Predictor with Graph Laplacian-Based Multi-task Learning

Tran

Khushi

2022

Computational Science – ICCS 2022

Self Cite

View full text Add to dashboard Cite

The stock market is a complex network that consists of individual stocks exhibiting various financial properties and different data distribution. For stock prediction, it is natural to build separate models for each stock but also consider the complex hidden correlation among a set of stocks. We propose a federated multi-task stock predictor with financial graph Laplacian regularization (FMSP-FGL). Specifically, we first introduce a federated multi-task framework with graph Laplacian regularization to fit separate but related stock predictors simultaneously. Then, we investigate the problem of graph Laplacian learning, which represents the association of the dynamic stock. We show that the proposed optimization problem with financial Laplacian constraints captures both the inter-series correlation between each pair of stocks and the relationship within the same stock cluster, which helps improve the predictive performance. Empirical results on two popular stock indexes demonstrate that the proposed method outperforms baseline approaches. To the best of our knowledge, this is the first work to utilize the advantage of graph Laplacian in multi-task learning for financial data to predict multiple stocks in parallel.

show abstract

A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

Cited by 38 publications

References 35 publications

A Greedy Agglomerative Framework for Clustered Federated Learning

A Greedy Agglomerative Framework for Clustered Federated Learning

PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

Stock Predictor with Graph Laplacian-Based Multi-task Learning

Contact Info

Product

Resources

About