Visual Representations for Semantic Target Driven Navigation

Abstract

What is a good visual representation for navigation? We study this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a previously unseen environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues.We propose to use semantic segmentation and detection masks as observations obtained by state-of-the-art computer vision algorithms and use a deep network to learn the navigation policy. The availability of equitable representations in simulated environments enables joint training using real and simulated data and alleviates the need for domain adaptation or domain randomization commonly used to tackle the sim-toreal transfer of the learned policies. Both the representation and the navigation policy can be readily applied to real nonsynthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach successfully gets to the target in 54% of the cases in unexplored environments, compared to 46% for a non-learning based approach, and 28% for a learning-based baseline.

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

“…Embodied Navigation Agent Navigation in 3D environments [58,29,30,16] is an essential capability of a mobile intelligent system that functions in the physical world. In the past two years, a plethora of tasks and evaluation protocols [38,25,40,52,3] have been proposed as summarized in [1].…”

Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks

“…Wu et al [46] use a gated-LSTM policy with multi-modal inputs trained for room navigation. Moursavian et al [31] use an LSTM policy for target driven navigation. The drawbacks of RNNs are mainly two-fold.…”

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

“…A similar problem has been studied in the context of mobility under problem domains of target-driven and semantic visual navigation [15,32,50]. These studies look at finding visual targets in unknown environments without maps through sensory pattern matching.…”