The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation

Zhao, Xiaoming; Agrawal, Harsh; Batra, Dhruv; Schwing, Alexander G.

doi:10.1109/iccv48922.2021.01582

Cited by 27 publications

(22 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, following prior work [11,37], we remove the assumption of oracle indoor localization and investigate our models under conditions of noisy pose. Specifically, we perturb the ground-truth pose sequence from our dataset in two ways.…”

Section: Resultsmentioning

confidence: 99%

“…We show qualitative results of a zero-shot transfer of our approach to a real-world, RGB-D dataset [29] that presents significantly challenging conditions of imperfect depth, pose and camera jitter -typical deployment conditions for egocentric AR assistants. In addition to that, we break away from the unrealistic assumption of the availability of oracle pose in indoor settings [11,37] and perform a systematic study of the impact of noise (of varying types and intensities) in the agent's pose. We show that our model is more resilient than baselines to such noisy perturbations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Episodic Memory Question Answering

Datta¹,

Dharur²,

Cartillier³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Egocentric augmented reality devices such as wearable glasses passively capture visual data as a human wearer tours a home environment. We envision a scenario wherein the human communicates with an AI agent powering such a device by asking questions (e.g., "where did you last see my keys?"). In order to succeed at this task, the egocentric AI assistant must (1) construct semantically rich and efficient scene memories that encode spatio-temporal information about objects seen during the tour and (2) possess the ability to understand the question and ground its answer into the semantic memory representation. Towards that end, we introduce (1) a new task -Episodic Memory Question Answering (EMQA) wherein an egocentric AI assistant is provided with a video sequence (the tour) and a question as an input and is asked to localize its answer to the question within the tour, (2) a dataset of grounded questions designed to probe the agent's spatio-temporal understanding of the tour, and (3) a model for the task that encodes the scene as an allocentric, top-down semantic feature map and grounds the question into the map to localize the answer. We show that our choice of episodic scene memory outperforms naive, off-the-shelf solutions for the task as well as a host of very competitive baselines and is robust to noise in depth, pose as well as camera jitter. The project page can be found at https://samyak-268.github.io/emqa/.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Episodic Memory Question Answering

Datta¹,

Dharur²,

Cartillier³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Relaxing supervisory sensors. There have been some attempts to relax the assumptions about perfect sensory information using visual odometry for the navigation task [10,56]. While these are effective approaches for PointGoal navigation, the same approach does not apply to manipulation as they still require the goal's location in the agent's initial coordinate frame.…”

Section: Related Workmentioning

confidence: 99%

Object Manipulation via Visual Target Localization

Ehsani¹,

Farhadi²,

Kembhavi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Object manipulation is a critical skill required for Embodied AI agents interacting with the world around them. Training agents to manipulate objects, poses many challenges. These include occlusion of the target object by the agent's arm, noisy object detection and localization, and the target frequently going out of view as the agent moves around in the scene. We propose Manipulation via Visual Object Location Estimation (m-VOLE), an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible, thus robustly aiding the task of manipulating these objects throughout the episode. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite but is trained without the object location estimator, and our analysis shows that our agent is robust to noise in depth perception and agent localization. Importantly, our proposed approach relaxes several assumptions about idealized lo-calization and perception that are commonly employed by recent works in embodied AI -an important step towards training agents for object manipulation in the real world.

show abstract

“…We assume that we have access to ground-truth pose as provided by the simulator to facilitate DD-PPO. We note that estimating the pose from noisy sensor readings is out of the scope of this work, and point to visual odometry methods [57] that can adapt DD-PPO agents to such a setting.…”

Section: Overview Of Our Approachmentioning

confidence: 99%

Cross-modal Map Learning for Vision and Language Navigation

Georgakis¹,

Schmeckpeper²,

Wanchoo³

et al. 2022

Preprint

View full text Add to dashboard Cite

We consider the problem of Vision-and-Language Navigation (VLN). The majority of current methods for VLN are trained end-to-end using either unstructured memory such as LSTM, or using cross-modal attention over the egocentric observations of the agent. In contrast to other works, our key insight is that the association between language and vision is stronger when it occurs in explicit spatial representations. In this work, we propose a cross-modal map learning model for vision-and-language navigation that first learns to predict the top-down semantics on an egocentric map for both observed and unobserved regions, and then predicts a path towards the goal as a set of waypoints. In both cases, the prediction is informed by the language through cross-modal attention mechanisms. We experimentally test the basic hypothesis that language-driven navigation can be solved given a map, and then show competitive results on the full VLN-CE benchmark.

show abstract

The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation

Cited by 27 publications

References 23 publications

Episodic Memory Question Answering

Episodic Memory Question Answering

Object Manipulation via Visual Target Localization

Cross-modal Map Learning for Vision and Language Navigation

Contact Info

Product

Resources

About