PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-Based Planning

Abstract

We present PRM-RL, a hierarchical method for long-range navigation task completion that combines samplingbased path planning with reinforcement learning (RL). The RL agents learn short-range, point-to-point navigation policies that capture robot dynamics and task constraints without knowledge of the large-scale topology. Next, the sampling-based planners provide roadmaps which connect robot configurations that can be successfully navigated by the RL agent. The same RL agents are used to control the robot under the direction of the planning, enabling long-range navigation. We use the Probabilistic Roadmaps (PRMs) for the sampling-based planner. The RL agents are constructed using feature-based and deep neural net policies in continuous state and action spaces. We evaluate PRM-RL, both in simulation and on-robot, on two navigation tasks with non-trivial robot dynamics: end-to-end differential drive indoor navigation in office environments, and aerial cargo delivery in urban environments with load displacement constraints. Our results show improvement in task completion over both RL agents on their own and traditional sampling-based planners. In the indoor navigation task, PRM-RL successfully completes up to 215 m long trajectories under noisy sensor conditions, and the aerial cargo delivery completes flights over 1000 m without violating the task constraints in an environment 63 million times larger than used in training.

Long-Range Indoor Navigation With PRM-RL

“…Overall, we show improved performance, better roadmap generation, and easier on-robot transfer, including a relative success rate increase of 40% over [21], and 94% over [11], while maintaining good performance despite increasing noise. We also show that only adding edges when agents can always navigate them makes roadmaps cheaper to build and improves navigation success; denser roadmaps also have higher simulated success rates but at substantial roadmap construction cost.…”

“…In other words, PRM-RL is a tool for generating paths which an RL agent can reliably satisfy without violating the constraints of its task. In [21] we demonstrated PRM-RL's success on tasks with constraints, but in this work, we focus solely on the navigation task, which collapses the task predicate L(x) to remaining within C f ree , and collapses the full configuration space available to the robot to a task space T limited to the robot's position and orientation.…”

“…The selected baselines are AutoRL [12], PRM-SL [34], PRM-GAPF (a modification of [11]), and our original PRM-RL with a hand-tuned RL policy [21]. The AutoRL [12] baseline policy is not guided by a PRM.…”

Towards Learning Abstract Representations for Locomotion Planning in High-dimensional State Spaces

“…Hence, it promising to combine these approaches and merge the advantages of both. Faust et al [15] use a reinforcement learning agent to learn short-range, point-to-point navigation policies for 2D and 3D action spaces which capture the robot dynamic and task constraint without considering the large-scale topology. Sampling-based planning is used to plan waypoints which give the planning a long-range goal-directed behavior.…”

Multi-Agent Motion Planning for Dense and Dynamic Environments via Deep Reinforcement Learning

“…A bad reward function may lead to local minima or may lead an agent to wander around forever. To overcome this issue in long-range navigation tasks [20] presents a hybrid approach that combines sampling based path planning with RL. [14], [15] adds a separate pre-step supervised learning(SL) phase to the RL approach to teach agents how to reach their goal location without caring about colliding with other agents.…”