Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Qi, Jiaju; Zhou, Qihao; Lei, Lei; Zheng, Kan

doi:10.48550/arxiv.2108.11887

Cited by 12 publications

(21 citation statements)

References 96 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are four key properties of federated learning: distribution, data protection, generality and status equality [Qi et al, 2021]. Distribution states that federated model training is done in parallel by all participating models.…”

Section: Federated Machine Learning and Federated Reinforcement Learningmentioning

confidence: 99%

“…As a result, the scale and diversity of data collection needed to achieve adequate performance quickly becomes infeasible for a single party. To account for these challenges, a common technique for data collection relies on crowdsourcing, where training episodes are collected from many agents that explore their local environment and then transmitted to a central controller to train a generalized model [Qi et al, 2021].…”

Section: Introductionmentioning

confidence: 99%

“…In reinforcement learning, policy and critic networks are already prone to instability, which could be further exacerbated by the averaging of model parameters. In addition, the averaging of model weights across multiple agents often prohibits individual exploration in non-identical environments, a challenge when federating models that explore diverse environments [Qi et al, 2021]. An example of this can be seen in autonomous vehicles, where knowledge gained from agents exploring rural roads is less useful to agents exploring urban cities.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FedFormer: Contextual Federation with Attention in Reinforcement Learning

Liam¹,

Golab²,

Poupart³

et al. 2022

Preprint

View full text Add to dashboard Cite

A core issue in federated reinforcement learning is defining how to aggregate insights from multiple agents into one. This is commonly done by taking the average of each participating agent's model weights into one common model (FedAvg). We instead propose FedFormer, a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings from models originating from different learner agents. In so doing, we attentively weigh contributions of other agents with respect to the current agent's environment and learned relationships, thus providing more effective and efficient federation 1 . We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor Critique single agent methods. Our results compared to Soft Actor Critique show that FedFormer performs better while still abiding by the privacy constraints of federated learning. In addition, we demonstrate nearly linear improvements in effectiveness with increased agent pools in certain tasks. This is contrasted by FedAvg, which fails to make noticeable improvements when scaled.1 Our code can be found at https://github.com/liamhebert/FedFormer Preprint. Under review.

show abstract

Section: Federated Machine Learning and Federated Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FedFormer: Contextual Federation with Attention in Reinforcement Learning

Liam¹,

Golab²,

Poupart³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are some works which study FL with an RL system [330]- [332]. Similar with FL [333], the idea of FRL could also be divided into two main categories: horizontal FRL (HFRL) and vertical FRL (VFRL) [334]. In HFRL, the mobile devices distribute geographically but face similar tasks.…”

Section: Federated Reinforcement Learningmentioning

confidence: 99%

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Yao¹,

Zhang²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Influenced by the great success of deep learning via cloud computing and the rapid development of edge chips, research in artificial intelligence (AI) has shifted to both of the computing paradigms, i.e., cloud computing and edge computing. In recent years, we have witnessed significant progress in developing more advanced AI models on cloud servers that surpass traditional deep learning models owing to model innovations (e.g., Transformers, Pretrained families), explosion of training data and soaring computing capabilities. However, edge computing, especially edge and cloud collaborative computing, are still in its infancy to announce their success due to the resource-constrained IoT scenarios with very limited algorithms deployed. In this survey, we conduct a systematic review for both cloud and edge AI. Specifically, we are the first to set up the collaborative learning mechanism for cloud and edge modeling with a thorough review of the architectures that enable such mechanism. We also discuss potentials and practical experiences of some on-going advanced edge AI topics including pretraining models, graph neural networks and reinforcement learning. Finally, we discuss the promising directions and challenges in this field.

show abstract

“…Furthermore, most of these works do not account for the underlying semantic relatedness of DRL tasks across agents and only rely on traditional similarity metrics (e.g., actionstate spaces) in MDP-based environments. In addition, most of existing federated RL (FRL) approaches (e.g., see [15] and references therein) involve all agents in CDRL without accounting for the similarities of DRL tasks, or for the resource constraints of the wireless network. Hence, there is a need for novel CDRL techniques that can address these problems.…”

Section: Introductionmentioning

confidence: 99%

Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks

Lotfi¹,

Semiari²,

Saad³

2021

Preprint

View full text Add to dashboard Cite

Collaborative deep reinforcement learning (CDRL) algorithms in which multiple agents can coordinate over a wireless network is a promising approach to enable future intelligent and autonomous systems that rely on real-time decision making in complex dynamic environments. Nonetheless, in practical scenarios, CDRL face many challenges due to heterogeneity of agents and their learning tasks, different environments, time constraints of the learning, and resource limitations of wireless networks. To address these challenges, in this paper, a novel semantic-aware CDRL method is proposed to enable a group of heterogeneous untrained agents with semantically-linked DRL tasks to collaborate efficiently across a resource-constrained wireless cellular network. To this end, a new heterogeneous federated DRL (HFDRL) algorithm is proposed to select the best subset of semantically relevant DRL agents for collaboration. The proposed approach then jointly optimizes the training loss and wireless bandwidth allocation for the cooperating selected agents in order to train each agent within the time limitation of its realtime task. Simulation results show the superior performance of the proposed algorithm compared to state-of-the-art baselines.

show abstract

Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

Cited by 12 publications

References 96 publications

FedFormer: Contextual Federation with Attention in Reinforcement Learning

FedFormer: Contextual Federation with Attention in Reinforcement Learning

Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI

Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks

Contact Info

Product

Resources

About