Collective Multiagent Sequential Decision Making Under Uncertainty

Lau

2021

ICAPS

Self Cite

Many real world systems involve interaction among large number of agents to achieve a common goal, for example, air traffic control. Several model-free RL algorithms have been proposed for such settings. A key limitation is that the empirical reward signal in model-free case is not very effective in addressing the multiagent credit assignment problem, which determines an agent's contribution to the team's success. This results in lower solution quality and high sample complexity. To address this, we contribute (a) an approach to learn a differentiable reward model for both continuous and discrete action setting by exploiting the collective nature of interactions among agents, a feature commonly present in large scale multiagent applications; (b) a shaped reward model analytically derived from the learned reward model to address the key challenge of credit assignment; (c) a model-based multiagent RL approach that integrates shaped rewards into well known RL algorithms such as policy gradient, soft-actor critic. Compared to previous methods, our learned reward models are more accurate, and our approaches achieve better solution quality on synthetic and real world instances of air traffic control, and cooperative navigation with large agent population.

Section: Learning System Reward Modelmentioning

confidence: 94%

Section: Introductionmentioning

confidence: 99%

Learning and Exploiting Shaped Reward Models for Large Scale Multiagent RL

Singh

Lau

2021

ICAPS

Self Cite

“…Our developed knowledge compilation methods are general and applicable in a variety of multiagent models, e.g., in collective Dec-POMDPs where agents are identical to each other (Nguyen, Kumar, and Lau 2017), or the heterogeneous setting where each agent is unique.…”

Section: The Dec-pomdp Model and Mapfmentioning

confidence: 99%

“…In cooperative sequential multiagent decision making, agents acting in a partially observable and uncertain environment are required to take coordinated decisions towards a long term goal (Durfee and Zilberstein 2013). Decentralized partially observable MDPs (Dec-POMDPs) provide a rich framework for multiagent planning (Bernstein et al 2002;Oliehoek and Amato 2016), and are applicable in domains such as vehicle fleet optimization (Nguyen, Kumar, and Lau 2017), cooperative robotics (Amato et al 2019), and multiplayer video games (Rashid et al 2018). However, scalability remains a key challenge with even a 2-agent Dec-POMDP, which is NEXP-Hard to solve optimally (Bernstein et al 2002).…”

Section: Introductionmentioning

confidence: 99%

Integrating Knowledge Compilation with Reinforcement Learning for Routes

Ling

Chandak

2021

ICAPS

Self Cite

Sequential multiagent decision-making under partial observability and uncertainty poses several challenges. Although multiagent reinforcement learning (MARL) approaches have increased the scalability, addressing combinatorial domains is still challenging as random exploration by agents is unlikely to generate useful reward signals. We address cooperative multiagent pathfinding under uncertainty and partial observability where agents move from their respective sources to destinations while also satisfying constraints (e.g., visiting landmarks). Our main contributions include: (1) compiling domain knowledge such as underlying graph connectivity and domain constraints into propositional logic based decision diagrams, (2) developing modular techniques to integrate such knowledge with deep MARL algorithms, and (3) developing fast algorithms to query the compiled knowledge for accelerated episode simulation in RL. Empirically, our approach can tractably represent various types of domain constraints, and outperforms previous MARL approaches significantly both in terms of sample complexity and solution quality on a number of instances.

Section: Introductionmentioning

confidence: 99%

Planning and Learning for Decentralized MDPs With Event Driven Rewards

Gupta

Paruchuri

2018

AAAI

Self Cite

Decentralized (PO)MDPs provide a rigorous framework for sequential multiagent decision making under uncertainty. However, their high computational complexity limits the practical impact. To address scalability and real-world impact, we focus on settings where a large number of agents primarily interact through complex joint-rewards that depend on their entire histories of states and actions. Such history-based rewards encapsulate the notion of events or tasks such that the team reward is given only when the joint-task is completed. Algorithmically, we contribute---1) A nonlinear programming (NLP) formulation for such event-based planning model; 2) A probabilistic inference based approach that scales much better than NLP solvers for a large number of agents; 3) A policy gradient based multiagent reinforcement learning approach that scales well even for exponential state-spaces. Our inference and RL-based advances enable us to solve a large real-world multiagent coverage problem modeling schedule coordination of agents in a real urban subway network where other approaches fail to scale.