On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning

Cohen, Andrew S.; Teng, Ervin; Berges, Vincent-Pierre; Dong, Ruo-Ping; Henry, Hunter; Mattar, Marwan; Zook, Alexander; Ganguly, Subha

doi:10.48550/arxiv.2111.05992

Cited by 9 publications

(16 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, since all elements that refine or de-refine are removed immediately and their individual trajectories terminate, they do not observe future states and cannot use future rewards to assign credit to their refine/de-refine actions. This is the posthumous multi-agent credit assignment problem [8]. We propose the use of centralized training to address this problem.…”

Section: Agent Creation and Deletionmentioning

confidence: 99%

“…In this work, we present the first formulation of AMR as a Markov game [25,18] and propose a novel fully-cooperative deep multi-agent reinforcement learning (MARL) algorithm [9,16,40] called Value Decomposition Graph Network (VDGN) to train a team of independently and simultaneously acting agents, each of which is a decision-maker for an element, to optimize a global performance metric and find anticipatory refinement policies. Because refinement and de-refinement actions at each step of the AMR Markov game leads to the creation and deletion of agents, we face the posthumous credit assignment problem [8]: agents who contributed to a future reward are not necessarily present at the future time to experience it. We show that VDGN, by virtue of centralized training with decentralized execution [24], addresses this key challenge.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement

Yang¹,

Mittal²,

Dzanic³

et al. 2022

Preprint

View full text Add to dashboard Cite

Adaptive mesh refinement (AMR) is necessary for efficient finite element simulations of complex physical phenomenon, as it allocates limited computational budget based on the need for higher or lower resolution, which varies over space and time. We present a novel formulation of AMR as a fully-cooperative Markov game, in which each element is an independent agent who makes refinement and de-refinement choices based on local information. We design a novel deep multi-agent reinforcement learning (MARL) algorithm called Value Decomposition Graph Network (VDGN), which solves the two core challenges that AMR poses for MARL: posthumous credit assignment due to agent creation and deletion, and unstructured observations due to the diversity of mesh geometries. For the first time, we show that MARL enables anticipatory refinement of regions that will encounter complex features at future times, thereby unlocking entirely new regions of the error-cost objective landscape that are inaccessible by traditional methods based on local error estimators. Comprehensive experiments show that VDGN policies significantly outperform error threshold-based policies in global error and cost metrics. We show that learned policies generalize to test problems with physical features, mesh geometries, and longer simulation times that were not seen in training. We also extend VDGN with multi-objective optimization capabilities to find the Pareto front of the tradeoff between cost and error.

show abstract

Section: Agent Creation and Deletionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement

Yang¹,

Mittal²,

Dzanic³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The episodic interaction of the two classes can be highlighted in Figure 5b. The Multi-Agent Posthumous Credit Assingment (MA-POCA) trainer was introduced by Unity Technologies (Cohen et al 2021). MA-POCA utilises the Independent Actor with Centralized Critic (IACC) framework, where a critic that is trained on joint information is utilized for updating independent agents or actors.…”

Section: Episodic Training Of Masmentioning

confidence: 99%

A Multi-Agent Reinforcement Learning Approach for System-Level Flight Delay Absorption

Malhotra

Lim

Alam

2022

2022 Winter Simulation Conference (WSC)

View full text Add to dashboard Cite

With increasing air traffic, there is an ever-growing need for Air Traffic Controllers (ATCO) to efficiently manage traffic and congestion. Congestion often leads to increased delays in the Terminal Maneuvering Area (TMA), causing large amounts of fuel burn and detrimental environmental impacts. Approaches such as the Extended Arrival Manager (E-AMAN) propose solutions to absorb such delays, whereby flights are scheduled much before they enter the TMA. However, such an approach requires a speed management system where flights can coordinate to absorb system-level delays in their en-route phase. This paper proposes a Multi-Agent System (MAS) approach using Deep Reinforcement Learning to model and train flights as agents which can coordinate with each other to effectively absorb system-level delays. The simulations utilize Multi-Agent POsthumous Credit Assignment in Unity and test two reward approaches. Initial findings reveal an average of 3.3 minutes of system-level delay absorptions from a required delay of 4 minutes. AUTHOR BIOGRAPHYKANUPRIYA MALHOTRA completed her B.Eng degree in Mechanical Engineering with a minor in Computing and Data Analytics. Her research experience involves working on optimizing routing and scheduling algorithms for postal delivery via Unmanned Aerial Vehicles in Singapore. She has contributed to researches done on the implementation of computer vision and artificial intelligence in applications such as automated waste sorting and autonomous underwater vehicles. She is currently working at Micron as a Data Analytic Engineer.

show abstract

“…Because the CTDE paradigm learns from multiple agents sharing their policies, it can learn the optimal policy faster and more effectively than a single agent. Multi-Agent Posthmous Credit Assignment (MA-POCA) proposed by Cohen et al [7] performs Fig. 2: Overview of MRC model: Following the CTDE method, critic collects the environment information observed by each user at each time a reset occurs, and determines the actions accordingly.…”

Section: Reinforcement Learningmentioning

confidence: 99%

“…Table 1: MRC model hyperparameters: γ and λ are discount factor and Temporal-Difference (TD) parameter in reinforcement learning [7].…”

Section: Actionmentioning

confidence: 99%

Optimal Planning for Redirected Walking Based on Reinforcement Learning in Multi-user Environment with Irregularly Shaped Physical Space

Lee

Cho

Min

et al. 2020

2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)

View full text Add to dashboard Cite

Fig. 1: Reset control during multi-user RDW experience in a physical space with a circle obstacle in the center: (a) Reset-to-Center (R2C): the user unconditionally reset to the center of the physical space without considering other users or obstacle in the center. (b) Reset-to-Gradient (R2G): the user reset to the direction of the sum of the force that is acted on them by the boundary of the physical space, obstacles, and other users. (c) Multi-user Reset Controller (MRC): the user reset to the optimal direction determined by a model trained with information about the user's physical space.

show abstract

On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning

Cited by 9 publications

References 9 publications

Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement

Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement

A Multi-Agent Reinforcement Learning Approach for System-Level Flight Delay Absorption

Optimal Planning for Redirected Walking Based on Reinforcement Learning in Multi-user Environment with Irregularly Shaped Physical Space

Contact Info

Product

Resources

About