FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization

Tarzanagh, Davoud Ataee; Li, Mingchen; Thrampoulidis, Christos; Oymak, Samet

doi:10.48550/arxiv.2205.02215

Cited by 4 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method and its analysis are standard and we include them in Section D.3 due to the limit of space. In Remark 1 in Section D.3, we also show that the complexity of our method is lower than those of [25,34].…”

Section: Federated Bilevel Optimization Algorithmmentioning

confidence: 62%

“…There are much fewer studies on BO in a distributed setting. The recent works [25,34] consider a BO where both the outer and inner problems are defined with the expectations over data distributed across nodes. They analyze the communication complexity of their methods in a non-convex setting.…”

Section: Related Workmentioning

confidence: 99%

“…They analyze the communication complexity of their methods in a non-convex setting. We apply a different FL algorithm based on Local-SVRG because our problem ( P) has a finite-sum structure that allows periodically going through all the data points in each node to obtain exact gradient information and achieving lower communication complexity than [25,34].…”

Section: Related Workmentioning

confidence: 99%

“…Remark 1. The federated bilevel optimization methods by [25,34] can find an ǫ-stationary point within Õ(ǫ −3 ) and Õ(ǫ −4 ) rounds of communication, respectively. In the first setting of Theorem 4 (τ = 1), Algorithm 3 has complexity of Õ(ǫ −2 ), which is better than [25,34].…”

Section: D3 Algorithm and Communication Complexity For Non-convex Fmentioning

confidence: 99%

“…The federated bilevel optimization methods by [25,34] can find an ǫ-stationary point within Õ(ǫ −3 ) and Õ(ǫ −4 ) rounds of communication, respectively. In the first setting of Theorem 4 (τ = 1), Algorithm 3 has complexity of Õ(ǫ −2 ), which is better than [25,34]. We want to point out that that the lower complexity of Algorithm 3 is because it utilizes the finite-sum structure in ( P), which allows computing a deterministic gradient infrequently to accelerate the convergence.…”

Section: D3 Algorithm and Communication Complexity For Non-convex Fmentioning

confidence: 99%

See 4 more Smart Citations

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

Huang¹,

Lin²,

Street³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose a federated learning method with weighted nodes in which the weights can be modified to optimize the model's performance on a separate validation set. The problem is formulated as a bilevel optimization where the inner problem is a federated learning problem with weighted nodes and the outer problem focuses on optimizing the weights based on the validation performance of the model returned from the inner problem. A communication-efficient federated optimization algorithm is designed to solve this bilevel optimization problem. Under an error-bound assumption, we analyze the generalization performance of the output model and identify scenarios when our method is in theory superior to training a model only locally and to federated learning with static and evenly distributed weights.

show abstract

Section: Federated Bilevel Optimization Algorithmmentioning

confidence: 62%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: D3 Algorithm and Communication Complexity For Non-convex Fmentioning

confidence: 99%

Section: D3 Algorithm and Communication Complexity For Non-convex Fmentioning

confidence: 99%

See 3 more Smart Citations

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

Huang¹,

Lin²,

Street³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

When Decentralized Optimization Meets Federated Learning

Gao¹,

Thai²,

Wu³

2023

IEEE Network

View full text Add to dashboard Cite

Federated learning is a new learning paradigm for extracting knowledge from distributed data. Due to its favorable properties in preserving privacy and saving communication costs, it has been extensively studied and widely applied to numerous data analysis applications. However, most existing federated learning approaches concentrate on the centralized setting, which is vulnerable to a single-point failure. An alternative strategy for addressing this issue is the decentralized communication topology. In this article, we systematically investigate the challenges and opportunities when renovating decentralized optimization for federated learning. In particular, we discussed them from the model, data, and communication sides, respectively, which can deepen our understanding about decentralized federated learning.

show abstract

Adaptive Federated Minimax Optimization with Lower complexities

Huang¹

2022

Preprint

View full text Add to dashboard Cite

Federated learning is a popular distributed and privacy-preserving machine learning approach. Meanwhile, minimax optimization is an effective hierarchical model in machine learning. Recently, some federated learning methods have been proposed to solve the distributed minimax optimization. However, these federated minimax optimization methods still suffer from high gradient and communication complexities. To fill this gap, in the paper, we study the Nonconvex-Strongly-Concave (NSC) minimax optimization, and propose a class of accelerated federated minimax optimization methods (i.e., FGDA and AdaFGDA) to solve the distributed minimax problems. Specifically, our methods build on the momentum-based variance reduced and local-SGD techniques, and our adaptive algorithm (i.e., AdaFGDA) can flexibly incorporate various adaptive learning rates by using the unified adaptive matrix. Theoretically, we provide a solid convergence analysis framework for our algorithms under non-i.i.d. setting. Moreover, we prove our algorithms obtain lower gradient (i.e., SFO) complexity of Õ(ǫ −3 ) with lower communication complexity of Õ(ǫ −2 ) in finding ǫ-stationary point of NSC minimax problems. Experimentally, we conduct the distributed fair learning and robust federated learning tasks to verify efficiency of our methods.

show abstract

FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization

Cited by 4 publications

References 0 publications

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

Federated Learning on Adaptively Weighted Nodes by Bilevel Optimization

When Decentralized Optimization Meets Federated Learning

Adaptive Federated Minimax Optimization with Lower complexities

Contact Info

Product

Resources

About