Learning Multi-Scene Absolute Pose Regression with Transformers

Abstract

Absolute camera pose regressors estimate the position and orientation of a camera given the captured image alone. Typically, a convolutional backbone with a multi-layer perceptron (MLP) head is trained using images and pose labels to embed a single reference scene at a time. Recently, this scheme was extended to learn multiple scenes by replacing the MLP head with a set of fully connected layers. In this work, we propose to learn multi-scene absolute camera pose regression with Transformers, where encoders are used to aggregate activation maps with selfattention and decoders transform latent features and scenes encoding into pose predictions. This allows our model to focus on general features that are informative for localization, while embedding multiple scenes in parallel. We extend our previous MS-Transformer approach [1] by introducing a mixed classificationregression architecture that improves the localization accuracy. Our method is evaluated on commonly benchmark indoor and outdoor datasets and has been shown to exceed both multi-scene and state-of-the-art single-scene absolute pose regressors. We make our code publicly available from here.

Leveraging Equivariant Features for Absolute Pose Regression

“…conclude that the proposed E-PoseNet achieves the lowest location error across all the outdoor and indoor scenes, and the lowest orientation error across the majority of them. It also competes with most recent transformer-based architectures [15,16] on these datasets.…”

Learning Selective Sensor Fusion for State Estimation

“…PoseNet has been further improved by combining CNNs and LSTMs for feature correlation [43], introducing temporal information [6], incorporating spatial constraints [1] or by adding additional covisibility constraints based on local maps and the estimated odometry [49]. MS-Transformer [36] is a recent relocalization work based on transformer architecture, achieving the stateof-the-art results.…”

“…We follow the official data split to train and test our models above this dataset. Task Baselines: Our SelectFusion model is built as an end-toend relocalization model, and thus we compare with LSTM-Pose [43], VidLoc [6], and MS-Transformer [36] which are representative within this category of learning techniques.…”

“…In addition to the common baselines, we also choose three representative visual localization approaches as task baselines, i.e. LSTM-Pose [43], VidLoc [6] and MS-Transformer [36]. VidLoc can be viewed as a simple direct fusion, but it uses full-size images, and different feature encoders.…”

MeshLoc: Mesh-Based Visual Localization

“…An alternative to explicitly representing the 3D scene geometry via a 3D model is to implicitly store information about the scene in the weights of a machine learning model. Examples include scene coordinate regression techniques [9, 10, 12, 14-16, 75, 90], which regress 2D-3D matches rather than computing them via explicit descriptor matching, and absolute [34,35,48,74,93] and relative pose [3,23,37] regressors. Scene coordinate regressors achieve state-ofthe-art results for small scenes [8], but have not yet shown strong performance in more challenging scenes.…”