Semi-parametric Topological Memory for Navigation

Savinov, Nikolay; Dosovitskiy, Alexey; Koltun, Vladlen

doi:10.48550/arxiv.1803.00653

Cited by 48 publications

(103 citation statements)

References 30 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…Parisotto & Salakhutdinov (2017) investigate a memory system to navigate in mazes. Some methods (Sepulveda et al, 2018;Chen et al, 2019;Savinov et al, 2018) use both visual features and the topological guidance of scenes for navigation, while natural-language instructions are employed to guide an agent to route among rooms (Anderson et al, 2018b;Wang et al, 2019;Deng et al, 2020;Hu et al, 2019;Majumdar et al, 2020;Hao et al, 2020). We notice that transformer architectures are also employed by Hao et al (2020) et al, 2015) and thus not the most prominent ones across the feature pyramid.…”

Section: Related Workmentioning

confidence: 93%

VTNet: Visual Transformer Network for Object Goal Navigation

Du¹,

Yu²,

Zheng³

2021

Preprint

View full text Add to dashboard Cite

Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions. In this paper, we introduce a Visual Transformer Network (VTNet) for learning informative visual representation in navigation. VTNet is a highly effective structure that embodies two key properties for visual representations: First, the relationships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional navigation signals can be learned. Furthermore, we also develop a pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In a nutshell, VTNet embeds object and region features with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations to achieve informative representation for navigation. Given such visual representations, agents are able to explore the correlations between visual observations and navigation actions. For example, an agent would prioritize "turning right" over "turning left" when the visual representation emphasizes on the right side of activation map. Experiments in the artificial environment AI2-Thor demonstrate that VTNet significantly outperforms state-of-the-art methods in unseen testing environments.

show abstract

Section: Related Workmentioning

confidence: 93%

VTNet: Visual Transformer Network for Object Goal Navigation

Du¹,

Yu²,

Zheng³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar issues also exist in [34]. [28] represents the agent's current state with respect to the goal state through a semi-parametric topological memory while it requires a pre-exploration stage to build a landmark graph. The authors of [10] locate the goal position in their predicted top-down egocentric free space map.…”

Section: Related Workmentioning

confidence: 97%

Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph

Yang

2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We present a novel two-layer hierarchical reinforcement learning approach equipped with a Goals Relational Graph (GRG) for tackling the partially observable goal-driven task, such as goal-driven visual navigation. Our GRG captures the underlying relations of all goals in the goal space through a Dirichlet-categorical process that facilitates: 1) the highlevel network raising a sub-goal towards achieving a designated final goal; 2) the low-level network towards an optimal policy; and 3) the overall system generalizing unseen environments and goals. We evaluate our approach with two settings of partially observable goal-driven tasks -a grid-world domain and a robotic object search task. Our experimental results show that our approach exhibits superior generalization performance on both unseen environments and new goals 1 .

show abstract

“…Combining RL and Planning. A few recent works have augmented RL-based methods with topological structures, like graphs [3,10,22,27,33] or memory buffers [4,15,20]. They store representations of the visited locations and exploit them at navigation time.…”

Section: Related Workmentioning

confidence: 99%

Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Mezghani,

Sukhbaatar,

Lavril

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we address the problem of image-goal navigation in the context of visually-realistic 3D environments. This task involves navigating to a location indicated by a target image in a previously unseen environment. Earlier attempts, including RL-based and SLAM-based approaches, have either shown poor generalization performance, or are heavily-reliant on pose/depth sensors. We present a novel method that leverages a cross-episode memory to learn to navigate. We first train a state-embedding network in a self-supervised fashion, and then use it to embed previously-visited states into an agent's memory. In order to avoid overfitting, we propose to use data augmentation on the RGB input during training. We validate our approach through extensive evaluations, showing that our data-augmented memory-based model establishes a new state of the art on the image-goal navigation task in the challenging Gibson dataset. We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.

show abstract

Semi-parametric Topological Memory for Navigation

Cited by 48 publications

References 30 publications

VTNet: Visual Transformer Network for Object Goal Navigation

VTNet: Visual Transformer Network for Object Goal Navigation

Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph

Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Contact Info

Product

Resources

About