Offline Visual Representation Learning for Embodied Navigation

Yadav, Karmesh; Ramrakhya, Ram; Majumdar, Arjun; Berges, Vincent-Pierre; Kuhar, Sachit; Batra, Dhruv; Baevski, Alexei; Maksymets, Oleksandr

doi:10.48550/arxiv.2204.13226

Cited by 5 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-Zero Experience Required (ZER) [18]: first trains an ImageNav agent composed of two ResNet-9 encoders for processing the goal-image and agent observations, and a policy network consisting Table 1: Zero-shot ObjectNav performance on Gibson [4], HM3D [20], and MP3D [8] validation. All methods use a single RGB sensor for agent observations except CoW [21], which also uses depth observations and OVRL [16], which uses GPS+Compass for ObjectNav. Our approach (ZSON) substantially improves on previous zero-shot methods and narrows the gap to SOTA fully-supervised methods such as OVRL [16], which is not zero-shot and provided for reference.…”

Section: Methodsmentioning

confidence: 99%

“…For reference, these gains are on par or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. On HM3D, our agent's zero-shot SPL matches a state-of-the-art ObjectNav method [16] that trains with direct supervision from 40k human demonstrations.…”

Section: Zero-shot Objectnavmentioning

confidence: 99%

“…At each timestep t, our agent receives an egocentric RGB observation v t and a goal representation s g v . The observation is processed by a ResNet-50 [29] encoder, which is pretrained on the Omnidata Starter Dataset (OSD) [30] using self-supervised learning (DINO [31]) following the pretraining recipe presented in OVRL [16]. The output from the ResNet-50 encoder is concatenated with the goal representation s g v and an embedding of the agent's previous action a t−1 and then passed to the policy network composed of a two-layer LSTM.…”

Section: Learning Semantic-goal Navigationmentioning

confidence: 99%

“…We train our SemanticNav agent with reinforcement learning (RL). During RL training, we use two data augmentation techniques: color jitter and random translation (adapted from [16]). Specifically, we train with DD-PPO [32] using a reward function proposed for ImageNav by Al-Halah et al [18]:…”

Section: Learning Semantic-goal Navigationmentioning

confidence: 99%

“…Given the name of an object (e.g., "chair"), ObjectNav involves exploring a 3D environment to find any instance of the object. The last few years have witnessed the development of new environments [2][3][4][5][6], annotated 3D scans [7][8][9], datasets of human demonstrations [10], and approaches for ObjectNav [11][12][13][14][15][16], cumulatively leading to strong progress. For instance, the entries in the annual Habitat challenge [17] have jumped from 6% success (DD-PPO baseline in 2020) to 53% success (top entry in ongoing 2022 Habitat Challenge public leaderboard).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

Majumdar¹,

Aggarwal²,

Devnani³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We present a scalable approach for learning open-world object-goal navigation (ObjectNav) -the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e.g., "find a sink"). Our approach is entirely zero-shot -i.e., it does not require ObjectNav rewards or demonstrations of any kind. Instead, we train on the image-goal navigation (ImageNav) task, in which agents find the location where a picture (i.e., goal image) was captured. Specifically, we encode goal images into a multimodal, semantic embedding space to enable training semantic-goal navigation (SemanticNav) agents at scale in unannotated 3D environments (e.g., HM3D). After training, SemanticNav agents can be instructed to find objects described in free-form natural language (e.g., "sink," "bathroom sink," etc.) by projecting language goals into the same multimodal, semantic embedding space. As a result, our approach enables open-world ObjectNav. We extensively evaluate our agents on three ObjectNav datasets (Gibson, HM3D, and MP3D) and observe absolute improvements in success of 4.2% -20.0% over existing zero-shot methods. For reference, these gains are similar or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. In an open-world setting, we discover that our agents can generalize to compound instructions with a room explicitly mentioned (e.g., "Find a kitchen sink") and when the target room can be inferred (e.g., "Find a sink and a stove"). * equal contribution Preprint. Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Zero-shot Objectnavmentioning

confidence: 99%

Section: Learning Semantic-goal Navigationmentioning

confidence: 99%

Section: Learning Semantic-goal Navigationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

Majumdar¹,

Aggarwal²,

Devnani³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Grounding Implicit Goal Description for Robot Indoor Navigation via Recursive Belief Update

Chen¹,

Zhao²,

Zhang³

2020

2020 3rd International Conference on Control and Robots (ICCR)

View full text Add to dashboard Cite

Object goal navigation aims to navigate an agent to locations of a given object category in unseen environments. Classical methods explicitly build maps of environments and require extensive engineering while lacking semantic information for object-oriented exploration. On the other hand, end-to-end learning methods alleviate manual map design and predict actions using implicit representations. Such methods, however, lack an explicit notion of geometry and may have limited ability to encode navigation history. In this work, we propose an implicit spatial map for object goal navigation. Our implicit map is recursively updated with new observations at each step using a transformer. To encourage spatial reasoning, we introduce auxiliary tasks and train our model to reconstruct explicit maps as well as to predict visual features, semantic labels and actions. Our method significantly outperforms the state of the art on the challenging MP3D dataset and generalizes well to the HM3D dataset. We successfully deploy our model on a real robot and achieve encouraging object goal navigation results in real scenes using only a few real-world demonstrations. Code, trained models and videos are available at https://www.di. ens.fr/willow/research/onav_rim/.

show abstract

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Pirinen¹,

Samuelsson²,

Backsund³

et al. 2023

Linköping Electronic Conference Proceedings

View full text Add to dashboard Cite

Climate-induced disasters are and will continue to be on the rise, and thus search-and-rescue (SAR) operations, where the task is to localize and assist one or several people who are missing, become increasingly relevant. In many cases the rough location may be known and a UAV can be deployed to explore a given, confined area to precisely localize the missing people. Due to time and battery constraints it is often critical that localization is performed as efficiently as possible. In this work we approach this type of problem by abstracting it as an aerial view goal localization task in a framework that emulates a SAR-like setup without requiring access to actual UAVs. In this framework, an agent operates on top of an aerial image (proxy for a search area) and is tasked with localizing a goal that is described in terms of visual cues. To further mimic the situation on an actual UAV, the agent is not able to observe the search area in its entirety, not even at low resolution, and thus it has to operate solely based on partial glimpses when navigating towards the goal. To tackle this task, we propose AiRLoc, a reinforcement learning (RL)-based model that decouples exploration (searching for distant goals) and exploitation (localizing nearby goals). Extensive evaluations show that AiRLoc outperforms heuristic search methods as well as alternative learnable approaches, and that it generalizes across datasets, e.g. to disaster-hit areas without seeing a single disaster scenario during training. We also conduct a proof-of-concept study which indicates that the learnable methods outperform humans on average. Code and models have been made publicly available at https://github.com/aleksispi/airloc.

show abstract

Offline Visual Representation Learning for Embodied Navigation

Cited by 5 publications

References 0 publications

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

Grounding Implicit Goal Description for Robot Indoor Navigation via Recursive Belief Update

Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescue

Contact Info

Product

Resources

About