Hybrid Policy Learning for Multi-Agent Pathfinding

Abstract

In this work we study the behavior of groups of autonomous vehicles, which are the part of the Internet of Vehicles systems. One of the challenging modes of operation of such systems is the case when the observability of each vehicle is limited and the global/local communication is unstable, e.g. in the crowded parking lots. In such scenarios the vehicles have to rely on the local observations and exhibit cooperative behavior to ensure safe and efficient trips. This type of problems can be abstracted to the socalled multi-agent pathfinding when a group of agents, confined to a graph, have to find collision-free paths to their goals (ideally, minimizing an objective function e.g. travel time). Widely used algorithms for solving this problem rely on the assumption that a central controller exists for which the full state of the environment (i.e. the agents current positions, their targets, configuration of the static obstacles etc.) is known and they can not be straightforwardly be adapted to the partially-observable setups. To this end, we suggest a novel approach which is based on the decomposition of the problem into the two sub-tasks: reaching the goal and avoiding the collisions. To accomplish each of this task we utilize reinforcement learning methods such as Deep Monte Carlo Tree Search, Q-mixing networks, and policy gradients methods to design the policies that map the agents' observations to actions. Next, we introduce the policy-mixing mechanism to end up with a single hybrid policy that allows each agent to exhibit both types of behavior -the individual one (reaching the goal) and the cooperative one (avoiding the collisions with other agents). We conduct an extensive empirical evaluation that shows that the suggested hybrid-policy outperforms standalone stat-ofthe-art reinforcement learning methods for this kind of problems by a notable margin.

A Comprehensive Review on Leveraging Machine Learning for Multi-Agent Path Finding

“…In addition, MAPPER integrates an evolutionary algorithm to enhance the refinement of agent policies. Another example of communicationfree approaches can be seen in studies by Skrynnik et al [69], [160], [161].…”

“…Open Question 9: How can effectively learned implicit communication minimize the need and overhead of explicit communication while achieving comparable outcomes? Studies such as [69], [161] demonstrate that state-of-theart decentralized behavior can be achieved by utilizing only local observations per agent without the need for explicit communication.…”

“…As shown in Table 1, most RL-based approaches depend on complex reward shaping schemes currently necessary for stable learning. However, it is also appealing to learn effective policies using a simple reward structure, like AlphaZero [51] and its adaptation to decentralization MAPF [161], eliminating manual parameter tuning.…”

Pathfinding in stochastic environments: learning vs planning

“…Target matrix : if the agent’s goal is inside the observation field, then there is 1 in the cell, where it is located, and 0 in other cells. If the target does not fall into the view, then it is projected onto the nearest cell of the observation field ( Skrynnik et al., 2021 ).…”

“…It is also worth noting a direction of research where planning algorithms are combined with reinforcement learning. In Skrynnik et al (2021) and Davydov et al (2021) , the authors train RL agents in a centralized (QMIX) and decentralized (PPO) way for solving multi-agent pathfinding tasks. The resulting RL policies are combined with a planning approach (MCTS), which leverages the resulting performance.…”