Deep Depth Estimation from Visual-Inertial SLAM

Abstract

This paper addresses the problem of learning to complete a scene's depth from sparse depth points and images of indoor scenes. Specifically, we study the case in which the sparse depth is computed from a visual-inertial simultaneous localization and mapping (VI-SLAM) system. The resulting point cloud has low density, it is noisy, and has nonuniform spatial distribution, as compared to the input from active depth sensors, e.g., LiDAR or Kinect. Since the VI-SLAM produces point clouds only over textured areas, we compensate for the missing depth of the low-texture surfaces by leveraging their planar structures and their surface normals which is an important intermediate representation. The pretrained surface normal network, however, suffers from large performance degradation when there is a significant difference in the viewing direction (especially the roll angle) of the test image as compared to the trained ones. To address this limitation, we use the available gravity estimate from the VI-SLAM to warp the input image to the orientation prevailing in the training dataset. This results in a significant performance gain for the surface normal estimate, and thus the dense depth estimates. Finally, we show that our method outperforms other state-of-the-art approaches both on training (ScanNet [1] and NYUv2 [2]) and testing (collected with Azure Kinect [3]) datasets.

Towards Real-Time Monocular Depth Estimation for Robotics: A Survey

“…The fused feature maps are passed to an encoder-decoder network to generate the output. Sartipi et al [141] use RGB images, learned surface normals and sparse depth from visual-inertial SLAM (VI-SLAM) to infer dense depth maps. Since the depth map from VI-SLAM is more sparse, a sparse-depth enrichment step is performed to increase its density.…”