Diagnosing Vision-and-Language Navigation: What Really Matters

Abstract

Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation. Results show that indoor navigation agents refer to both object and direction tokens when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. Transformer-based agents acquire a better cross-modal understanding of objects and display strong numerical reasoning ability than non-Transformer-based agents. When it comes to vision-and-language alignments, many models claim that they can align object tokens with specific visual targets. We find unbalanced attention on the vision and text input and doubt the reliability of such cross-modal alignments. 1

Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas

“…While task completion significantly drops when direction tokens are masked, the agent still performs on a high level. This finding is surprising and in dissent with Zhu et al (2021a) who report that task completion nearly drops to zero when masking direction tokens during testing only. We believe that in our setting (masking during testing and training), the model learns to infer the correct directions from redundancies in the instructions or context around the direction tokens.…”

“…To analyze the importance of direction and object tokens in the navigation instructions, we run masking experiments similar to Zhu et al (2021a), except that we mask the tokens during training and testing instead of during testing only. Figure 4 shows the resulting task completion rates for an increasing number of masked direction or object tokens.…”

“…Agents in this outdoor environment are trained to follow human written navigation instructions (Chen et al, 2019;Xiang et al, 2020), instructions generated by Google Maps (Hermann et al, 2020), or a combination of both (Zhu et al, 2021b). Recent work focuses on analyzing the navigation agents by introducing better trajectory overlap metrics Ilharco et al, 2019) or diagnosing the performance under certain constraints such as uni-modal inputs (Thomason et al, 2019) and masking direction or object tokens (Zhu et al, 2021a). Other work used a trained VLN agent to evaluate automatically generated navigation instructions (Zhao et al, 2021).…”

Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions

“…High-level features such as visual appearance, route structure, and detected objects outperform the low level visual features extracted by CNN (Hu et al, 2019). Different types of tokens within the instruction also function differently (Zhu et al, 2021b). Extracting these tokens and encoding the object tokens and directions tokens are crucial Zhu et al, 2021b).…”

“…Different types of tokens within the instruction also function differently (Zhu et al, 2021b). Extracting these tokens and encoding the object tokens and directions tokens are crucial Zhu et al, 2021b).…”