Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?

Abstract

Accurate visual localization is a key technology for autonomous navigation. 3D structure-based methods employ 3D models of the scene to estimate the full 6 degree-of-freedom (DOF) pose of a camera very accurately. However, constructing (and extending) large-scale 3D models is still a significant challenge. In contrast, 2D image retrieval-based methods only require a database of geo-tagged images, which is trivial to construct and to maintain. They are often considered inaccurate since they only approximate the positions of the cameras. Yet, the exact camera pose can theoretically be recovered when enough relevant database images are retrieved. In this paper, we demonstrate experimentally that large-scale 3D models are not strictly necessary for accurate visual localization. We create reference poses for a large and challenging urban dataset. Using these poses, we show that combining image-based methods with local reconstructions results in a higher pose accuracy compared to state-of-the-art structure-based methods, albeight at higher run-time costs. We show that some of these run-time costs can be alleviated by exploiting known database image poses. Our results suggest that we might want to reconsider the need for large-scale 3D models in favor of more local models, but also that further research is necessary to accelerate the local reconstruction process.

VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change

“…It is important to note here that image matching can also be included as a sub-module of a VPR system. Torii et al (2019) demonstrated that such a system can achieve accurate localisation without the need for large-scale 3D-models.…”

“…McManus et al (2014) extract scene signatures from an image by utilising some a priori environment information and describe them using HOG-descriptors. DenseVLAD presented by Torii et al (2015) is a Vector-of-Locally-Aggregated-Descriptorsbased approach using densely sampled SIFT keypoints, which has been shown to perform similar to deep-learningbased techniques in Sattler et al (2018) and Torii et al (2019). A more recent usage of traditional handcrafted feature descriptors for VPR was presented in CoHOG (Zaffar et al 2020) which focuses on entropy-rich regions in an image and uses HOG as the regional descriptor for convolutionalregional matching.…”

“…The benchmarks have usually been organised as challenges (which have their own dedicated utility), where relevant evaluation papers also exist, e.g. the recent detailed works from Torii et al (2019) and Sattler et al (2018). Google also proposed the Landmarks dataset with focus on both place/instance-level recognition and retrieval: Google Landmark V1 dataset (Noh et al 2017) and Google Landmark V2 dataset (Weyand et al 2020).…”

A Survey on Deep Visual Place Recognition

“…One problem with using 3D models is that they are expensive to store and maintain, and do not scale well to large environments. To overcome this problem [214] proposes to not use a large 3D model created offline, but rather to create a small local model online (SfMon-the-fly), using the top retrieved images from the database. This solution is shown to achieve good results, although its effectiveness deteriorates in the case of weakly textured scenes.…”

“…A first broad categorization of these datasets comes from the distinction between robotics and non-robotics datasets. Robotics datasets [21], [45], [151], [152], [214], [240], [241], [247]- [249], [249] are typically created by recording videos from cameras mounted on a vehicle (e.g., car) or a smaller robot. The data is thus available in sequences, with the temporal coherence among successive frames that can be leveraged to formulate motion hypotheses.…”