Expert Sample Consensus Applied to Camera Re-Localization

Abstract

Fitting model parameters to a set of noisy data points is a common problem in computer vision. In this work, we fit the 6D camera pose to a set of noisy correspondences between the 2D input image and a known 3D environment. We estimate these correspondences from the image using a neural network. Since the correspondences often contain outliers, we utilize a robust estimator such as Random Sample Consensus (RANSAC) or Differentiable RANSAC (DSAC) to fit the pose parameters. When the problem domain, e.g. the space of all 2D-3D correspondences, is large or ambiguous, a single network does not cover the domain well. Mixture of Experts (MoE) is a popular strategy to divide a problem domain among an ensemble of specialized networks, so called experts, where a gating network decides which expert is responsible for a given input. In this work, we introduce Expert Sample Consensus (ESAC), which integrates DSAC in a MoE. Our main technical contribution is an efficient method to train ESAC jointly and end-to-end. We demonstrate experimentally that ESAC handles two real-world problems better than competing methods, i.e. scalability and ambiguity. We apply ESAC to fitting simple geometric models to synthetic images, and to camera re-localization for difficult, real datasets.

Hierarchical Scene Coordinate Classification and Regression for Visual Localization

“…Instead of learning the en-tire pipeline, scene coordinate regression methods learn the first stage of the pipeline in the structure-based approaches. Namely, either a random forest [4,12,13,20,30,32,33,50,57] or a neural network [3,5,6,7,9,10,11,27,28,30] is trained to directly predict 3D scene coordinates for the pixels and thus the 2D-3D correspondences are established. These methods do not explicitly rely on feature detection, description and matching, and are able to provide correspondences densely.…”

“…These methods do not explicitly rely on feature detection, description and matching, and are able to provide correspondences densely. They are more accurate than traditional feature-based methods at small and medium scale, but usually do not scale well to larger scenes [5,6]. In order to generalize well from novel viewpoints, these methods typically rely on only local image patches to produce the scene coordinate predictions.…”

“…In [9], scene coordinate regression is formulated as two separate tasks of object instance recognition and local coordinate regression. In [6], multiple scene coordinate regression networks are trained as a mixture of experts along with a gating network which assesses the relevance of each expert for a given input, and the final pose estimate is obtained using a novel RANSAC framework, i.e., Expert Sample Consensus (ESAC). In contrast to existing approaches, in our work, we use spatially dense discrete location labels, i.e.…”

“…Note that we address single-image RGB localization, as in e.g. [5,6,7,28], rather than using RGB-D images [11,12,13,20,33,50,57] or image sequences [37,55].…”

“…Ground truth poses and sparse 3D reconstructions generated with structure from motion are also provided. In addition to these three datasets, we synthesize three largescale indoor scenes based on 7-Scenes and 12-Scenes by placing all seven, twelve or nineteen individual scenes, into a single coordinate system similar to [6]. These large integrated datasets are denoted by i7-Scenes (i7S), i12-Scenes (i12S), i19-Scenes (i19S), respectively.…”