PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning

Abstract

Multi-agent path finding (MAPF) is an essential component of many large-scale, real-world robot deployments, from aerial swarms to warehouse automation. However, despite the community's continued efforts, most state-of-the-art MAPF planners still rely on centralized planning and scale poorly past a few hundred agents. Such planning approaches are maladapted to real-world deployments, where noise and uncertainty often require paths be recomputed online, which is impossible when planning times are in seconds to minutes. We present PRIMAL, a novel framework for MAPF that combines reinforcement and imitation learning to teach fully-decentralized policies, where agents reactively plan paths online in a partially-observable world while exhibiting implicit coordination. This framework extends our previous work on distributed learning of collaborative policies by introducing demonstrations of an expert MAPF planner during training, as well as careful reward shaping and environment sampling. Once learned, the resulting policy can be copied onto any number of agents and naturally scales to different team sizes and world dimensions. We present results on randomized worlds with up to 1024 agents and compare success rates against state-of-theart MAPF planners. Finally, we experimentally validate the learned policies in a hybrid simulation of a factory mockup, involving both real-world and simulated robots.

A survey on multi-robot coverage path planning for model reconstruction and mapping

“…Also, based on the cooperation method used whether it's a centralized, distributed or decentralized, the team of robots need to maintain communication especially if the team shares information. Most of the reviewed work assumes perfect communication and utilizes centralized type of cooperation as presented in [3,20,44,70] which is subject to scalability, overhead and single point of failure problems. Some of the work presented a decentralized CPP approach for multi-robot systems as presented in [8,52,54,70].…”

“…The common assumption in literature is that communication is unlimited in range and bandwidth. Nearly all centralized systems assume that the individual robots can communicate directly with the central controller such as [44,70], and algorithms that create maps assume global communication [53,57]. Since communication systems are often down in the aftermath of a disaster [51,61], achieving coverage with limited communication must be a critical aspect that must to be considered.…”

“…The distributed partitioning is performed by incrementing pseudo sensor range in steps while the range constrained Voronoi cell gets contracted to obtain the desired region. The work in [70] proposed a hybrid decentralized multiagent path finding framework. It combines Reinforcement Learning (RL) for planning single agent paths and Imitation learning from centralized path planner.…”

Clone Swarms: Learning to Predict and Control Multi-Robot Systems by Imitation

“…In addition to imitation learning, modern reinforcement learning methods have also been used to generate MRS behavior [11]. The approach in [12] combines both reinforcement learning and imitation learning for extracting multi-agent navigation policies. The approach can deal with partially-observable domains, variable team-sizes, as well as complex environments and mazes.…”

A method for real-time dynamic fleet mission planning for autonomous mining

“…Since the vehicles share the same environment, their coordination [13] and cooperation [14] is central when solving the planning problem. The methods presented in [14,15], and [16] are examples of multi-vehicle path planning algorithms that use artificial intelligence-based approaches (primarily reinforcement learning), to solve the problem of planning in multi-vehicle systems, demonstrating strong scalability properties [16].…”