Large-scale, real-time visual–inertial localization revisited

Abstract

<jats:p> The overarching goals in image-based localization are scale, robustness, and speed. In recent years, approaches based on local features and sparse 3D point-cloud models have both dominated the benchmarks and seen successful real-world deployment. They enable applications ranging from robot navigation, autonomous driving, virtual and augmented reality to device geo-localization. Recently, end-to-end learned localization approaches have been proposed which show promising results on small-scale datasets. However, the positioning accuracy, scalability, latency, and compute and storage requirements of these approaches remain open challenges. We aim to deploy localization at a global scale where one thus relies on methods using local features and sparse 3D models. Our approach spans from offline model building to real-time client-side pose fusion. The system compresses the appearance and geometry of the scene for efficient model storage and lookup leading to scalability beyond what has been demonstrated previously. It allows for low-latency localization queries and efficient fusion to be run in real-time on mobile platforms by combining server-side localization with real-time visual–inertial-based camera pose tracking. In order to further improve efficiency, we leverage a combination of priors, nearest-neighbor search, geometric match culling, and a cascaded pose candidate refinement step. This combination outperforms previous approaches when working with large-scale models and allows deployment at unprecedented scale. We demonstrate the effectiveness of our approach on a proof-of-concept system localizing 2.5 million images against models from four cities in different regions of the world achieving query latencies in the 200 ms range. </jats:p>

Using Image Sequences for Long-Term Visual Localization

“…However, in many other applications, e.g., autonomous driving, the cameras record image sequences rather than taking individual pictures. There is a significant body of work on using sequences for visual localization [23,28,35,54,55,58,61,71,84,91]. These works combine single-image localization against a pre-built scene representation with local camera tracking based on visual and/or inertial odometry.…”

“…Inspired by [28,51], this coarse estimate is used as a prior for a single-image localization module that provides pose estimates with respect to a global map. Finally, a fine localization stage similar to [35,54,55,58,61,84] refines the pose information provided by the single-image algorithm.…”

The UMA-SAR Dataset: Multimodal data collection from a ground vehicle during outdoor disaster response training exercises

“…Liu et al (2018) and Dubé et al (2020) addressed 3D lidar-based SLAM methods on natural terrains and semi-unstructured environments, respectively, but not in collapsed, cluttered, and unstructured outdoor environments where assumptions such as segments and planes are not valid. Recent works on largescale and real-time visual SLAM show high-quality maps and loop-closing in outdoor urban environments (Lynen et al, 2020;Tanner et al, 2020), where GPS and 3D lidar are used as ground truth. Moreover, works in urban environments using vision, semantic mapping (Cadena et al, 2016), and including TIR images (Shin and Kim, 2019) offer promising techniques for potential SLAM applications in disaster robotics.…”

Image‐Based Indoor Localization Using Smartphone Camera

“…In the training phase, the calculative scale is very large, and GPU servers are usually required, which cannot run smoothly on mobile platforms [20]. In many scenarios, learning-based features are not as effective as traditional features such as SIFT, and the interpretability is poor [24][25][26][27].…”