Let's Take This Online: Adapting Scene Coordinate Regression Network Predictions for Online RGB-D Camera Relocalisation

Abstract

Many applications require a camera to be relocalised online, without expensive offline training on the target scene. Whilst both keyframe and sparse keypoint matching methods can be used online, the former often fail away from the training trajectory, and the latter can struggle in textureless regions. By contrast, scene coordinate regression (SCoRe) methods generalise to novel poses and can leverage dense correspondences to improve robustness, and recent work has shown how to adapt SCoRe forests between scenes, allowing their state-of-the-art performance to be leveraged online. However, because they use features hand-crafted for indoor use, they do not generalise well to harder outdoor scenes. Whilst replacing the forest with a neural network and learning suitable features for outdoor use is possible, the techniques used to adapt forests between scenes are unfortunately harder to transfer to a network context. In this paper, we address this by proposing a novel way of leveraging a network trained on one scene to predict points in another scene. Our approach replaces the appearance clustering performed by the branching structure of a regression forest with a two-step process that first uses the network to predict points in the original scene, and then uses these predicted points to look up clusters of points from the new scene. We show experimentally that our online approach achieves state-of-the-art performance on both the 7-Scenes and Cambridge Landmarks datasets, whilst running in under 300ms, making it highly effective in live scenarios.

Using Image Sequences for Long-Term Visual Localization

“…Another recent line of work on single-image localization has focused on machine learning [7,8,10,11,14,15,37,38,57,75,86,88]. Scene coordinate regression approaches [7,8,14,15,57,75,88] train a random forest or convolutional neural network (CNN) to predict the corresponding 3D coordinate for each pixel.…”

“…The 2D-3D matches are then used for camera pose estimation, e.g., by applying a PnP solver [1,31,41,43,45,46] inside a robust estimator such as RANSAC [4,9,17,25,47,66]. These visual localization methods typically use either local image descriptors [19,22,34,52] to explicitly match 2D features to 3D scene points or use machine learning, e.g., via a random forest [15,16] or a convolutional neural network (CNN) [6,7,14], to regress the corresponding 3D scene coordinate per pixel. They build a scene representation, e.g., a 3D Structure-from-Motion (SfM) model for local features or a CNN for scene coordinate regression, from a set of reference images.…”

Hierarchical Scene Coordinate Classification and Regression for Visual Localization

“…Active Search [43] and an indoor localization method which exploits dense correspondences [53]. Note that, in general, methods that exploit additional depth information [11,12] Figure 3. Average pose accuracy on the combined scenes.…”

“…Instead of learning the en-tire pipeline, scene coordinate regression methods learn the first stage of the pipeline in the structure-based approaches. Namely, either a random forest [4,12,13,20,30,32,33,50,57] or a neural network [3,5,6,7,9,10,11,27,28,30] is trained to directly predict 3D scene coordinates for the pixels and thus the 2D-3D correspondences are established. These methods do not explicitly rely on feature detection, description and matching, and are able to provide correspondences densely.…”

“…Note that we address single-image RGB localization, as in e.g. [5,6,7,28], rather than using RGB-D images [11,12,13,20,33,50,57] or image sequences [37,55].…”