Understanding the Limitations of CNN-Based Absolute Camera Pose Regression

Abstract

Visual localization is the task of accurate camera pose estimation in a known scene. It is a key problem in computer vision and robotics, with applications including selfdriving cars, Structure-from-Motion, SLAM, and Mixed Reality. Traditionally, the localization problem has been tackled using 3D geometry. Recently, end-to-end approaches based on convolutional neural networks have become popular. These methods learn to directly regress the camera pose from an input image. However, they do not achieve the same level of pose accuracy as 3D structure-based methods. To understand this behavior, we develop a theoretical model for camera pose regression. We use our model to predict failure cases for pose regression techniques and verify our predictions through experiments. We furthermore use our model to show that pose regression is more closely related to pose approximation via image retrieval than to accurate pose estimation via 3D structure. A key result is that current approaches do not consistently outperform a handcrafted image retrieval baseline. This clearly shows that additional research is needed before pose regression algorithms are ready to compete with structure-based methods.

Lending Orientation to Neural Networks for Cross-View Geo-Localization

“…While previous image-based geolocalization methods have been primarily based on ground-to-ground (streetview level) image matching (e.g., [5,24,19,18,20,27]), using ground-to-aerial cross-view matching (i.e., matching ground-level photos to aerial imagery ) is becoming an attractive approach for image-based localization, thanks to the widespread coverage (over the entire Earth) and easy accessibility of satellite and aerial survey images. In contrast, the coverage of ground-level street views (such as Google-map or Bing-map) is at best limited to urban areas.…”

Let's Take This Online: Adapting Scene Coordinate Regression Network Predictions for Online RGB-D Camera Relocalisation

“…Like local matching methods, they also generalise well from novel poses. However, whilst they tend to be more accurate than local matching methods at small/medium scale, they have not yet been shown to scale well to very large scenes [60].…”

Satellite Pose Estimation with Deep Landmark Regression and Nonlinear Pose Refinement

“…While such end-toend methods have demonstrated some success, they have not achieved similar accuracy as geometry-based solutions (e.g., those that optimise pose from a correspondence set). Moreover, recent work [29] suggests that "absolute pose regression approaches are more closely related to approximate pose estimation via image retrieval", thus they may not generalise well in practice.…”