A Dual Semantic-Aware Recurrent Global-Adaptive Network for Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) is a realistic but challenging task that requires an agent to locate the target region using verbal and visual cues. While significant advancements have been achieved recently, there are still two broad limitations: (1) The explicit information mining for significant guiding semantics concealed in both vision and language is still under-explored; (2) The previously structured map method provides the average historical appearance of visited nodes, while it ignores distinctive contributions of various images and potent information retention in the reasoning process. This work proposes a dual semantic-aware recurrent global-adaptive network (DSRG) to address the above problems. First, DSRG proposes an instruction-guidance linguistic module (IGL) and an appearance-semantics visual module (ASV) for boosting vision and language semantic learning respectively. For the memory mechanism, a global adaptive aggregation module (GAA) is devised for explicit panoramic observation fusion, and a recurrent memory fusion module (RMF) is introduced to supply implicit temporal hidden states. Extensive experimental results on the R2R and REVERIE datasets demonstrate that our method achieves better performance than existing methods. Code is available at https://github.com/CrystalSixone/DSRG.

RES-StS: Referring Expression Speaker via Self-Training With Scorer for Goal-Oriented Vision-Language Navigation

“…Our model shows superior performance across multiple metrics. Notably, compared with the previous state-of-the-art DSRG [29], CausalVLN gains significant improvements in SR (↑ 2.61%) and RGS (↑ 2.13%) on the validation unseen set. Similar enhancements are also observed on the validation seen and test unseen sets.…”

“…Finally, we utilize the memory-augmented global-local crossmodal fusion module from our previous work DSRG [29] to enable the agent to align and leverage features from different modalities, capturing valuable historical cues throughout the navigation.…”

“…SEvol [45] utilized the graph to construct relationships of objects. Our previous work DSRG [29] proposed a dual semantic-augmented module to model the semantics explicitly. Although previous methods have highlighted the importance of data augmentation and the leverage of semantic cues, we argue that these approaches are necessary but insufficient.…”

“…Therefore, it is essential to accurately represent the agent's state. In this paper, we employ the memory-augmented global-local crossmodal fusion proposed by our previous work [29], consisting of a global adaptive aggregation (GAA) method, a crossmodal encoder, and a recurrent memory fusion (RMF). GAA computes the visual representation at the t-th step as:…”

“…We adopt a similar configuration to DSRG [29], with 9 layers for the language encoder, 2 layers for the vision encoder, and 4 layers for the cross-modal encoder. To intervene with the object confounders, we use the object features extracted by the bottom-up attention model provided by [16].…”