Expanding the Limits of Vision‐based Localization for Long‐term Route‐following Autonomy

Abstract

Vision‐based, autonomous, route‐following algorithms enable robots to autonomously repeat manually driven routes over long distances. Through the use of inexpensive, commercial vision sensors, these algorithms have the potential to enable robotic applications across multiple industries. However, in order to extend these algorithms to long‐term autonomy, they must be able to operate over long periods of time. This poses a difficult challenge for vision‐based systems in unstructured and outdoor environments, where appearance is highly variable. While many techniques have been developed to perform localization across extreme appearance change, most are not suitable or untested for vision‐in‐the‐loop systems such as autonomous route following, which requires continuous metric localization to keep the robot driving. In this paper, we present a vision‐based, autonomous, route‐following algorithm that combines multiple channels of information during localization to increase robustness against daily appearance change such as lighting. We explore this multichannel visual teach and repeat framework by adding the following channels of information to the basic single‐camera, gray‐scale, localization pipeline: images that are resistant to lighting change and images from additional stereo cameras to increase the algorithm's field of view. Using these methods, we demonstrate robustness against appearance change through extensive field deployments spanning over 26 km with an autonomy rate greater than 99.9%. We furthermore discuss the limits of this system when subjected to harsh environmental conditions by investigating keypoint match degradation through time.

How to Train a CAT: Learning Canonical Appearance Transformations for Direct Visual Localization Under Illumination Change

“…Our direct localization pipeline operates in both mapping (VO) and relocalization modes in a similar vein to topometric visual teach-and-repeat navigation [13], [14], where the camera follows a similar trajectory during both mapping and relocalization phases. As the camera explores the environment in mapping mode, we generate a list of posed keyframes with corresponding image and depth data, creating new keyframes when the translational or rotational distance between the most recent keyframe pose and the current tracking pose exceeds a preset threshold.…”

“…To this end, we investigated the use of our trained CAT models for teach-and-repeat-style metric relocalization [13], [14] by first creating a map in the canonical condition, then relocalizing against it in different conditions using both the original and transformed images. Figure 6 shows sample relocalization errors for the "Morning" and "Sunset" conditions of the VKITTI/0001 trajectory using both the original image streams (resized and cropped to 256 × 192 resolution) and the output of our trained model, while Table II summarizes our relocalization results for all ETHL/syn and VKITTI sequences.…”

Estimation and navigation methods with limited information for autonomous urban driving

“…The problems of navigating without GPS and a highly detailed map can be addressed separately in a less computationally expensive manner. Techniques have been proposed in recent years based on accurate self‐localization in mapped environments in an attempt to reduce the vehicle's reliance upon GPS . However, these methods still rely heavily on a priori map information, as they require either a lidar or vision sensor layer to be included in the environmental map.…”

“…Techniques have been proposed in recent years based on accurate self-localization in mapped environments in an attempt to reduce the vehicle's reliance upon GPS. [20][21][22][23][24] However, these methods still rely heavily on a priori map information, as they require either a lidar or vision sensor layer to be included in the environmental map. Likewise, sufficient localization in an extended GPS blackout can be provided by a posterior pose algorithm, which augments GPS and an inertial navigation system with vision-based measurements of nearby lanes and stop lines referenced to a known map of environmental features.…”

Visual‐inertial curve simultaneous localization and mapping: Creating a sparse structured world without feature points

“…The work in Ref. presents an algorithm that is invariant to lighting conditions, but it relies on feature points and requires multiple stereo cameras. The work in Refs.…”