SC-wLS: Towards Interpretable Feed-forward Camera Re-localization

Abstract

Visual re-localization aims to recover camera poses in a known environment, which is vital for applications like robotics or augmented reality. Feed-forward absolute camera pose regression methods directly output poses by a network, but suffer from low accuracy. Meanwhile, scene coordinate based methods are accurate, but need iterative RANSAC post-processing, which brings challenges to efficient end-to-end training and inference. In order to have the best of both worlds, we propose a feed-forward method termed SC-wLS that exploits all scene coordinate estimates for weighted least squares pose regression. This differentiable formulation exploits a weight network imposed on 2D-3D correspondences, and requires pose supervision only. Qualitative results demonstrate the interpretability of learned weights. Evaluations on 7Scenes and Cambridge datasets show significantly promoted performance when compared with former feed-forward counterparts. Moreover, our SC-wLS method enables a new capability: self-supervised test-time adaptation on the weight network. Codes and models are publicly available.

Improving Feature-based Visual Localization by Geometry-Aided Matching

“…End-to-End Localization. End-to-End localization methods can be classified as either directly regressing the camera pose [45], [46], [47], [48], [49] or regressing scene coordinates [50], [51], [52], [53]. They do not suffer from the problems of local feature matching and are very robust under the scenes with illumination changes or texture-less regions.…”