A Critical Investigation of Deep Reinforcement Learning for Navigation

Abstract

The navigation problem is classically approached in two steps: an exploration step, where mapinformation about the environment is gathered; and an exploitation step, where this information is used to navigate efficiently. Deep reinforcement learning (DRL) algorithms, alternatively, approach the problem of navigation in an end-to-end fashion. Inspired by the classical approach, we ask whether DRL algorithms are able to inherently explore, gather and exploit map-information over the course of navigation. We build upon Mirowski et al.[2017]'s work and introduce a systematic suite of experiments that vary three parameters: the agent's starting location, the agent's target location, and the maze structure. We choose evaluation metrics that explicitly measure the algorithm's ability to gather and exploit map-information. Our experiments show that when trained and tested on the same maps, the algorithm successfully gathers and exploits map-information. However, when trained and tested on different sets of maps, the algorithm fails to transfer the ability to gather and exploit map-information to unseen maps. Furthermore, we find that when the goal location is randomized and the map is kept static, the algorithm is able to gather and exploit map-information but the exploitation is far from optimal. We open-source our experimental suite in the hopes that it serves as a framework for the comparison of future algorithms and leads to the discovery of robust alternatives to classical navigation methods.

CLAMGen: Closed-Loop Arm Motion Generation via Multi-view Vision-Based RL

“…A big challenge with applying learning-based methods to path-planning is non-generalizability; [20] highlighted the difficulty of learned methods to generalize to the full path planning problem where the environment obstacles, start and goal locations are completely randomized. Because of this limitation, hybrid methods are developed to combine learning-based methods with conventional sampling based methods [11] [13].…”

Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goals

“…Goal directed visual navigation There has been considerable interest in using Deep Reinforcement Learning (DRL) algorithms for the goal-driven visual navigation of robots (Mirowski et al, 2016(Mirowski et al, , 2017Dhiman et al, 2018;Gupta et al, 2017;Savinov, Dosovitskiy, and Koltun, 2018). Mirowski et al (2016) demonstrate that a DRL algorithm called Asynchronous Advantage Actor Critic (A3C) can learn to find a goal in 3D navigation simulators, using only a front facing first person view as input, while Mirowski et al (2017) demonstrate goal directed navigation in Google's street view graph.…”

“…Moving the successes of these works from simulations to the real world is an active area of research because of the high sample complexity of model-free RL algorithms (Zhu et al, 2017;Anderson et al, 2018) . Dhiman et al (2018) empirically evaluate Mirowski et al (2016)'s approach and show that when goal locations are dynamic, the path chosen to reach the goal are often far from optimal. In contrast to our method, these works focus on the navigation domain and employ domain specific auxiliary rewards and data-structures making them less generalizable to other multi-goal tasks.…”

“…Distance-Inefficiency The distance-inefficiency (Dhiman et al, 2018) is the ratio of the distances travelled by the agent during an episode to the sum of the shortest path to the goal at every point of initialization. Mathematically it is defined as:…”

“…In multi-goal tasks that involve static environments and dynamic goals, such as goal-directed navigation and pickand-place in robotics, model-free RL struggles to transfer learned behavior from one goal location to another within the same environment (Dhiman et al, 2018;Quillen et al, 2018). This occurs because model-free RL represents the environment as value functions, which conflate the statedynamics and reward distributions into a single representation.…”