Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory

Abstract

The role of robots in society keeps expanding, bringing with it the necessity of interacting and communicating with humans. In order to keep such interaction intuitive, we provide automatic wayfinding based on verbal navigational instructions. Our first contribution is the creation of a large-scale dataset with verbal navigation instructions. To this end, we have developed an interactive visual navigation environment based on Google Street View; we further design an annotation method to highlight mined anchor landmarks and local directions between them in order to help annotators formulate typical, human references to those. The annotation task was crowdsourced on the AMT platform, to construct a new Talk2Nav dataset with 10, 714 routes. Our second contribution is a new learning method. Inspired by spatial cognition research on the mental conceptualization of navigational instructions, we introduce a soft dual attention mechanism defined over the segmented language instructions to jointly extract two partial instructions—one for matching the next upcoming visual landmark and the other for matching the local directions to the next landmark. On the similar lines, we also introduce spatial memory scheme to encode the local directional transitions. Our work takes advantage of the advance in two lines of research: mental formalization of verbal navigational instructions and training neural network agents for automatic way finding. Extensive experiments show that our method significantly outperforms previous navigation methods. For demo video, dataset and code, please refer to our project page.

Vision-Language Navigation: A Survey and Taxonomy

“…Talk to Nav. Vasudevan et al [148] developed an interactive visual navigation environment based on Google Street View named Talk2Nav dataset with 10,714 routes, and an built effective model to create large-scale navigational instructions over long-range city environments. 16 http://streetlearn.cc Street View.…”

Construction of a Visual Saliency Model for Neighborhood Building Landmarks Based on K‐Means Clustering

“…ere are a large number of variants of the K-means algorithm, including initialization optimization K-means++, distance calculation optimization Elkan K-means algorithm, and optimization Mini Batch K-means algorithm in the case of big data. e deterministic algorithm converts the landmark visual saliency problem into an optimization problem and converts the local pattern matching in the landmark visual saliency of the video sequence into a cost function minimization problem, the most representative deterministic algorithm is the K-means clustering algorithm, and the advantages of the K-means clustering algorithm are fast convergence, being used in the landmark visual saliency of the high frame rate, and being very suitable for the landmark visual saliency analysis of real-time scenes when a considerable number of landmark visual saliency algorithms are based on the improvement of K-means clustering algorithm [8]. However, the K-means clustering method also has its drawbacks; for example, it is difficult to cope with the scale change and shape change of the target in the landmark visual saliency model acquisition process, easy to be influenced by the similar background and the interference of light change, and easy to occur in the building surface clustering in the building surface process.…”

“…e artificial ant in the ant colony algorithm uses the overall information of the ant colony, and the global update of the residual pheromone is performed only after the completion of an optimization search. e pheromone update formula on each path in the ant colony algorithm is (8), where D(j) min is the intraclass distance when the objective function obtains the minimum value. v(wkj) is the pheromone increment; M is the total amount of pheromone released by ants; and D(j) (0 < D(j) < 1) is the pheromone volatility coefficient.…”

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

“…In recent years, researchers have investigated systems where passengers can give commands to self-driving cars. For instance, (Vasudevan, Dai, and Van Gool 2021;Chen et al 2019) consider navigational commands such as "Take the first left and at the red building turn right. Afterwards, drive to the white building".…”