Reinforcement Learning for Zone Based Multiagent Pathfinding under Uncertainty

Abstract

We address the problem of multiple agents finding their paths from respective sources to destination nodes in a graph (also called MAPF). Most existing approaches assume that all agents move at fixed speed, and that a single node accommodates only a single agent. Motivated by the emerging applications of autonomous vehicles such as drone traffic management, we present zone-based path finding (or ZBPF) where agents move among zones, and agents' movements require uncertain travel time. Furthermore, each zone can accommodate multiple agents (as per its capacity). We also develop a simulator for ZBPF which provides a clean interface from the simulation environment to learning algorithms. We develop a novel formulation of the ZBPF problem using difference-of-convex functions (DC) programming. The resulting approach can be used for policy learning using samples from the simulator. We also present a multiagent credit assignment scheme that helps our learning approach converge faster. Empirical results in a number of 2D and 3D instances show that our approach can effectively minimize congestion in zones, while ensuring agents reach their final destinations.

Integrating Knowledge Compilation with Reinforcement Learning for Routes

“…MARL for MAPF: MAPF can be mapped to a Dec-POMDP instance in multiple ways to address different variants (Ma, Kumar, and Koenig 2017;Sartoretti et al 2019;Ling, Gupta, and Kumar 2020). We present the MAPF problem under uncertainty and partial observability using minimal assumptions to ensure the generality of our knowledge compilation framework.…”

“…We have integrated our knowledge compilation framework with two policy gradient approaches proposed in (Sartoretti et al 2019;Ling, Gupta, and Kumar 2020) (one using feedforward neural net, another using recurrent neural network based policy), and a QMIX-variant (Fu et al 2019) for MAPF, demonstrating the generalization power of the framework for a range of MARL solution methods.…”

“…Even the deterministic MAPF setting where multiple agents need to find collision-free paths to their destinations in a shared environment is NP-Hard (Yu and LaValle 2013). The MAPF problem has important applications in several domains such as warehouse logistics (Wurman, D'Andrea, and Mountz 2008), robotics (Sartoretti et al 2019), and vehicle fleet optimization (Ling, Gupta, and Kumar 2020).…”

“…Deep MARL approaches have been applied to MAPF under uncertainty and partial observability (Sartoretti et al 2019;Ling, Gupta, and Kumar 2020). A key challenge is that it takes several simulations to find even a single route to destination as model-free RL does not explicitly exploit the underlying graph connectivity.…”

“…As the number of paths increases exponentially with the graph size, we also use hierarchical decomposition of the graph to enable a tractable sdd representation (Choi, Shen, and Darwiche 2017). We integrate our framework with previous MARL approaches (Sartoretti et al 2019;Ling, Gupta, and Kumar 2020), and show that the resulting algorithms significantly outperform the original algorithms both in terms of sample complexity and solution quality on a number of instances. We also compare environment simulation speed using our inference algorithm, and previous approaches based on model counting and specialized graph heuristics.…”