Target-driven visual navigation in indoor scenes using deep reinforcement learning

Abstract

Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new target goals, and (2) data inefficiency i.e., the model requires several (and often costly) episodes of trial and error to converge, which makes it impractical to be applied to real-world scenarios. In this paper, we address these two issues and apply our model to the task of target-driven visual navigation. To address the first issue, we propose an actor-critic model whose policy is a function of the goal as well as the current state, which allows to better generalize. To address the second issue, we propose AI2-THOR framework, which provides an environment with highquality 3D scenes and physics engine. Our framework enables agents to take actions and interact with objects. Hence, we can collect a huge number of training samples efficiently.We show that our proposed method (1) converges faster than the state-of-the-art deep reinforcement learning methods, (2) generalizes across targets and across scenes, (3) generalizes to a real robot scenario with a small amount of fine-tuning (although the model is trained in simulation), (4) is end-to-end trainable and does not need feature engineering, feature matching between frames or 3D reconstruction of the environment.The supplementary video can be accessed at the following link: https://youtu.be/SmBxMDiOrvs. arXiv:1609.05143v1 [cs.CV]

DroNet: Learning to Fly by Driving

“…Another promising approach has been use simulations to get training data for reinforcement or imitation learning tasks, while testing the learned policy in the real world [14], [15], [11]. Clearly, this approach suffers from the domain shift between simulation and reality and might require some realworld data to be able to generalize [11].…”

“…These methodologies can be divided into two main categories: (i) methods based on reinforcement learning (RL) [7], [11] and (ii) methods based on supervised learning [6], [12], [9], [10], [13].…”

Deep reward shaping from demonstrations

“…Asynchronous parallel sampling is applied to a number of deep reinforcement learning algorithms including DQN and an actor-critic method (called A3C) which achieved state of the art results on the Atari benchmark and demonstrated effective performance in a navigation task in a 3D simulated environment. Utilizing additional information about the target is used to speed up and improve deep RL in [11]. Information about the target is provided in the form of images of the target.…”

“…For comparison, in our experiments on a 30x30 Grid, the shortest route to the target consists of 57 actions. While a recent study proposed a new complex simulator with realistic graphics [11] where the shortest trajectory to the target is typically less than 20 actions. …”

“…Recently several successful attempts combine the use of deep learning with RL to learn a wide array of tasks such as Atari games [5][6], optimal control [7][8], board and card games [9][10] and navigation [11]. Most efforts use deep learning to directly map raw state representations to action space.…”