Beyond Controlled Environments: 3D Camera Re-Localization in Changing Indoor Scenes

Abstract

Long-term camera re-localization is an important task with numerous computer vision and robotics applications. Whilst various outdoor benchmarks exist that target lighting, weather and seasonal changes, far less attention has been paid to appearance changes that occur indoors. This has led to a mismatch between popular indoor benchmarks, which focus on static scenes, and indoor environments that are of interest for many real-world applications. In this paper, we adapt 3RScan -a recently introduced indoor RGB-D dataset designed for object instance re-localization -to create RIO10, a new long-term camera re-localization benchmark focused on indoor scenes. We propose new metrics for evaluating camera re-localization and explore how state-of-the-art camera re-localizers perform according to these metrics. We also examine in detail how different types of scene change affect the performance of different methods, based on novel ways of detecting such changes in a given RGB-D frame. Our results clearly show that long-term indoor re-localization is an unsolved problem. Our benchmark and tools are publicly available at waldjohannau.github.io/RIO10.

Towards Accurate Active Camera Localization

“…However, the 3D scene models in these datasets are either missing or constructed in the format of sparse point cloud, which is not suitable to render high-quality visual observations at any unpredictable position for reinforcement learning. On the other side, the indoor datasets in the past cover both room-level environments [45,49,51] and large-scale university buildings [48]. The constructed training and test scene meshes in these scanned real datasets mostly do not fully overlap with each other, and sometimes lack important geometric details, such as walls, which are however required to obtain a complete full-frame visual observation for real-time rendering in reinforcement learning.…”

“…In the implementation, we employ Cycles to render the photorealistic images, and the real-time renderer in Interactive Gibson [53] to synthesize camera images for reinforcement learning. For the real indoor scene, the texture is realistic in nature, but the constructed scene meshes tend to have some artifacts due to the limited scanner in the existing visual localization datasets [45,49,51]. We take advantage of the Matterport scanner device along with its scene reconstruction algorithms to obtain the high-quality complete indoor scene mesh, and achieve real-time rendering also in Interactive Gibson.…”