Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication

Su, Jianyu; Adams, Stephen; Beling, Peter A.

doi:10.48550/arxiv.2004.00470

Cited by 7 publications

(13 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is the reason why they did not implement a RL-based approach, but instead a semi-supervised learning algorithm, therefore guiding the learning process with explicit labeled data. In fact, so far the very few works that combine GNNs with a MARL framework [51], [52] are theoretical papers from the ML community, and none of them apply to the field of networking.…”

Section: Related Workmentioning

confidence: 99%

MAGNNETO: A Graph Neural Network-Based Multi-Agent System for Traffic Engineering

Bernárdez¹,

Suárez-Varela²,

López³

et al. 2023

IEEE Trans. Cogn. Commun. Netw.

View full text Add to dashboard Cite

Current trends in networking propose the use of Machine Learning (ML) for a wide variety of network optimization tasks. As such, many efforts have been made to produce ML-based solutions for Traffic Engineering (TE), which is a fundamental problem in ISP networks. Nowadays, state-of-the-art TE optimizers rely on traditional optimization techniques, such as Local search, Constraint Programming, or Linear programming. In this paper, we present MAGNNETO, a distributed ML-based framework that leverages Multi-Agent Reinforcement Learning and Graph Neural Networks for distributed TE optimization. MAGNNETO deploys a set of agents across the network that learn and communicate in a distributed fashion via message exchanges between neighboring agents. Particularly, we apply this framework to optimize link weights in OSPF, with the goal of minimizing network congestion. In our evaluation, we compare MAGNNETO against several state-of-the-art TE optimizers in more than 75 topologies (up to 153 nodes and 354 links), including realistic traffic loads. Our experimental results show that, thanks to its distributed nature, MAGNNETO achieves comparable performance to state-of-the-art TE optimizers with significantly lower execution times. Moreover, our ML-based solution demonstrates a strong generalization capability to successfully operate in new networks unseen during training.

show abstract

Section: Related Workmentioning

confidence: 99%

MAGNNETO: A Graph Neural Network-Based Multi-Agent System for Traffic Engineering

Bernárdez¹,

Suárez-Varela²,

López³

et al. 2023

IEEE Trans. Cogn. Commun. Netw.

View full text Add to dashboard Cite

show abstract

“…In summary, GCN exquisitely designs a structure to extract graph embedding. In MARL, there are some works (Ryu, Shin, and Park 2020;Jiang et al 2018;Su, Adams, and Beling 2020;Mao et al 2020) using GCN to encode the observations of agents to obtain a richer representation to help make simultaneous decisions.…”

Section: Graph Convolutional Networkmentioning

confidence: 99%

Learning Multi-Agent Action Coordination via Electing First-Move Agent

Ruan

Meng

Xiong

et al. 2022

ICAPS

View full text Add to dashboard Cite

Learning to coordinate actions among agents is essential in complicated multi-agent systems. Prior works are constrained mainly by the assumption that all agents act simultaneously, and asynchronous action coordination between agents is rarely considered. This paper introduces a bi-level multi-agent decision hierarchy for coordinated behavior planning. We propose a novel election mechanism in which we adopt a graph convolutional network to model the interaction among agents and elect a first-move agent for asynchronous guidance. We also propose a dynamically weighted mixing network to effectively reduce the misestimation of the value function during training. This work is the first to explicitly model the asynchronous multi-agent action coordination, and this explicitness enables to choose the optimal first-move agent. The results on Cooperative Navigation and Google Football demonstrate that the proposed algorithm can achieve superior performance in cooperative environments. Our code is available at https://github.com/Amanda-1997/EFA-DWM.

show abstract

“…QTRAN (Son et al 2019) is a generalized factorization method that can be applied to environments that are free from structural constraints. Other works, such as CommNet (Foerster et al 2016), TarMAC (Das et al 2019), ATOC (Jiang and Lu 2018), MAAC (Iqbal and Sha 2019), CCOMA (Su, Adams, and Beling 2020) and BiCNet (Peng et al 2017) exploit interagent communication.…”

Section: Related Workmentioning

confidence: 99%

Value-Decomposition Multi-Agent Actor-Critics

Adams

Beling

2021

AAAI

Self Cite

View full text Add to dashboard Cite

The exploitation of extra state information has been an active research area in multi-agent reinforcement learning (MARL). QMIX represents the joint action-value using a non-negative function approximator and achieves the best performance on the StarCraft II micromanagement testbed, a common MARL benchmark. However, our experiments demonstrate that, in some cases, QMIX performs sub-optimally with the A2C framework, a training paradigm that promotes algorithm training efficiency. To obtain a reasonable trade-off between training efficiency and algorithm performance, we extend value-decomposition to actor-critic methods that are compatible with A2C and propose a novel actor-critic framework, value-decomposition actor-critic (VDAC). We evaluate VDAC on the StarCraft II micromanagement task and demonstrate that the proposed framework improves median performance over other actor-critic methods. Furthermore, we use a set of ablation experiments to identify the key factors that contribute to the performance of VDAC.

show abstract

Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication

Cited by 7 publications

References 13 publications

MAGNNETO: A Graph Neural Network-Based Multi-Agent System for Traffic Engineering

MAGNNETO: A Graph Neural Network-Based Multi-Agent System for Traffic Engineering

Learning Multi-Agent Action Coordination via Electing First-Move Agent

Value-Decomposition Multi-Agent Actor-Critics

Contact Info

Product

Resources

About