Long-Term Visual Localization Using Semantically Segmented Images

Abstract

Robust cross-seasonal localization is one of the major challenges in long-term visual navigation of autonomous vehicles. In this paper, we exploit recent advances in semantic segmentation of images, i.e., where each pixel is assigned a label related to the type of object it represents, to attack the problem of long-term visual localization. We show that semantically labeled 3D point maps of the environment, together with semantically segmented images, can be efficiently used for vehicle localization without the need for detailed feature descriptors (SIFT, SURF, etc.). Thus, instead of depending on hand-crafted feature descriptors, we rely on the training of an image segmenter. The resulting map takes up much less storage space compared to a traditional descriptor based map. A particle filter based semantic localization solution is compared to one based on SIFT-features, and even with large seasonal variations over the year we perform on par with the larger and more descriptive SIFT-features, and are able to localize with an error below 1 m most of the time.

CoHOG: A Light-Weight, Compute-Efficient, and Training-Free Visual Place Recognition Technique for Changing Environments

“…Authors in [47], [48] have formulated visual localisation as a two-stage process: 1) global matching-based, less-intensive place matching candidates selection 2) local features-based, intensive final candidate selection with focus on spatial constraints. Other interesting approaches to place recognition have also been adopted, including semantic segmentation-based visual localisation (as in [49], [50], [51]) and object proposalsbased place recognition [52].…”

Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization

“…The latter type of methods have recently been shown to not perform consistently better than image retrieval methods [76], i.e., approaches that approximate the pose of the query image by the pose of the most similar database image [3,38,87]. As such, state-of-the-art methods for long-term visual localization at scale either rely on local features for matching [28,71,78,83,85,86] or use image retrieval techniques [2-4, 63, 80, 87, 94].…”

“…Particle Filter-based Semantic Localization (PFSL) [83]. In this approach, localization is approached as a filtering problem where we, in addition to a sequence of camera images, also have access to noisy odometry information.…”

“…To this end, we compare against HF-Net [72], a CNNbased hierarchical localization approach, Asymmetric Hypercolumn Matching [29], an approach based on matching of hypercolumn features, DenseVLAD [87], a stateof-the-art image retrieval pipeline, and its trainable variant NetVLAD [3], City Scale Localization [84], a non-semantic approach based on 2D-3D matches, GSMC [86] using the semantic segmentation network from [86], and PFSL [83] using semantic segmentation network from [42].…”

I Can See Clearly Now: Image Restoration via De-Raining

“…In this work we develop a filter which as a pre-processing step removes the effect of raindrops on lenses. Several tasks are affected by the presence of adherent water droplets on camera lenses or enclosures, such as semantic segmentation [1], localisation using segmentation [2], [3] or road marking segmentation [4]. In this paper we choose to use segmentation as an example task by which to test the effectiveness of our method.…”