Is This the Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization

Abstract

Visual localization in large and complex indoor scenes, dominated by weakly textured rooms and repeating geometric patterns, is a challenging problem with high practical relevance for applications such as Augmented Reality and robotics. To handle the ambiguities arising in this scenario, a common strategy is, first, to generate multiple estimates for the camera pose from which a given query image was taken. The pose with the largest geometric consistency with the query image, e.g., in the form of an inlier count, is then selected in a second stage. While a significant amount of research has concentrated on the first stage, there is considerably less work on the second stage. In this paper, we thus focus on pose verification. We show that combining different modalities, namely appearance, geometry, and semantics, considerably boosts pose verification and consequently pose accuracy. We develop multiple hand-crafted as well as a trainable approach to join into the geometric-semantic verification and show significant improvements over stateof-the-art on a very challenging indoor dataset.

Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis

“…This paper thus proposes a semi-automated approach to reference pose generation. Our method is inspired by previous work on pose verification via view synthesis (Taira et al 2018(Taira et al , 2019Torii et al 2018) and the observation that modern learned local features Revaud et al 2019) capture higher-level shape information. The latter allows feature matching between real images and 3D models, e.g., obtained via multi-view stereo .…”

“…State-of-the-art approaches for long-term localization Sarlin et al 2019;Germain et al 2019;Larsson et al 2019;Stenborg et al 2018;Yang et al 2020;Benbihi et al 2019;Taira et al 2018Taira et al , 2019 are based on local features and explicit 3D scene models. 1 Classical handcrafted features such as ORB (Rublee et al 2011), SIFT (Lowe 2004), and SURF (Bay et al 2008) struggle to match features between images taken under strongly differing viewing conditions, e.g., day and night or seasonal changes.…”

“…Besides using learned features that are more robust to changes in viewing conditions, long-term localization approaches also use semantic image segmentation (Budvytis et al 2019;Garg et al 2019;Larsson et al 2019;Stenborg et al 2018;Schönberger et al 2018;Seymour et al 2019;Shi et al 2019;Taira et al 2019;Toft et al 2017Toft et al , 2018Wang et al 2019;Yu et al 2018). These methods are based on the observation that the semantic meaning of scene elements, in contrast to their appearance, is invariant to changes.…”

“…These methods are based on the observation that the semantic meaning of scene elements, in contrast to their appearance, is invariant to changes. Semantic image segmentations are thus used as an invariant representation for image retrieval (Arandjelović and Zisserman 2014;Toft et al 2017;Yu et al 2018), to verify 2D-3D matches (Budvytis et al 2019;Larsson et al 2019;Stenborg et al 2018;Toft et al 2018) and camera pose estimates (Shi et al 2019;Stenborg et al 2018;Taira et al 2019;Toft et al 2018), for learning local features (Garg et al 2019;Schönberger et al 2018), and as an additional input to learning-based localization approaches (Budvytis et al 2019;Seymour et al 2019;Wang et al 2019).…”

“…Thus, the contributions of this paper center around a detailed experimental evaluation of the use of view synthesis to improve pose accuracy rather than on proposing a new method. Taira et al (2018Taira et al ( , 2019 use view synthesis for automated pose verification. To this end, they render a dense laser scan point cloud from a set of given poses.…”