Night-to-Day Image Translation for Retrieval-based Localization

Abstract

Visual localization is a key step in many robotics pipelines, allowing the robot to (approximately) determine its position and orientation in the world. An efficient and scalable approach to visual localization is to use image retrieval techniques. These approaches identify the image most similar to a query photo in a database of geo-tagged images and approximate the query's pose via the pose of the retrieved database image. However, image retrieval across drastically different illumination conditions, e.g. day and night, is still a problem with unsatisfactory results, even in this age of powerful neural models. This is due to a lack of a suitably diverse dataset with true correspondences to perform end-to-end learning. A recent class of neural models allows for realistic translation of images among visual domains with relatively little training data and, most importantly, without ground-truth pairings.In this paper, we explore the task of accurately localizing images captured from two traversals of the same area in both day and night. We propose ToDayGAN -a modified imagetranslation model to alter nighttime driving images to a more useful daytime representation. We then compare the daytime and translated night images to obtain a pose estimate for the night image using the known 6-DOF position of the closest day image. Our approach improves localization performance by over 250% compared the current state-of-the-art, in the context of standard metrics in multiple categories.

Geometrically Mappable Image Features

“…A recent work [21] proposes a method to improve mining of triplets composing of hard negatives for training. A few works have addressed the problem of seasonal or day-night variations either by using 3D point clouds [16] or by domain transfer [13]. Others have proposed better or faster matching [22], [23], facilitating image retrieval.…”

“…Moreover, it also means that precise localization may not be possible with such a feature embedding. Although several methods have shown accurate localization by learning features from densely distributed images [11], [12], [13], this may not always be feasible due to high computational and memory requirements. We argue that learning features whose distances are directly proportional to the geometric counterpart in the map, results in more versatile and powerful features which also provide higher retrieval accuracy.…”

Don’t Worry About the Weather: Unsupervised Condition-Dependent Domain Adaptation

“…Style-Transfer: Other approaches attempt to directly train computer vision models using synthetic data generated via style-transfer, or to directly adapt the input data to the target domain. Notable approaches include those of [4], [5], [23], [3] and [2]. Generally, these methods seem to have the most promise of reducing the domain gap between real and synthetic images, hence our decision to generate training data using the approach of [24].…”

“…As a type of domain adaptation technique, domain unification is the holy grail of visual perception, theoretically allowing models trained on samples with limited heterogeneity to perform adequately on scenes that are well out of the distribution of the training data. Domain unification can be applied within the vast distribution of natural images [1], [2], [3], between natural and synthetic images (computer-generated, whether through traditional 3D rendering or more modern GAN-based techniques) [4], [5] and even between different sensor modalities [6]. Additionally, domain unification can be implemented at different stages of a computer vision pipeline, ranging from direct approaches such as domain confusion [7], [8], [9], fine-tuning models on target domains [1] or mixture-of-expert approaches [10], etc.…”

Retrieval-based Localization Based on Domain-invariant Feature Learning under Changing Environments

“…Lowry et al [2] proposed a simple approach based on using modified PCA to remove dimensions of variant conditions and showed impressive results. Adversia Porav et al [3] and Anoosheh et al [4] both overcame condition variance through image translation. Yin et al [5] proposed to separate condition-invariant features from extracted features using a CNN.…”