Deep Surface Normal Estimation With Hierarchical RGB-D Fusion

Abstract

The growing availability of commodity RGB-D cameras has boosted the applications in the field of scene understanding. However, as a fundamental scene understanding task, surface normal estimation from RGB-D data lacks thorough investigation. In this paper, a hierarchical fusion network with adaptive feature re-weighting is proposed for surface normal estimation from a single RGB-D image. Specifically, the features from color image and depth are successively integrated at multiple scales to ensure global surface smoothness while preserving visually salient details. Meanwhile, the depth features are re-weighted with a confidence map estimated from depth before merging into the color branch to avoid artifacts caused by input depth corruption. Additionally, a hybrid multi-scale loss function is designed to learn accurate normal estimation given noisy ground-truth dataset. Extensive experimental results validate the effectiveness of the fusion strategy and the loss design, outperforming state-of-the-art normal estimation schemes.

Superb Monocular Depth Estimation Based on Transfer Learning and Surface Normal Guidance

“…Therefore, a confidence map network branch was set following the method proposed by Zeng et al [ 11 ], which generates confidence maps to indicate whether side effects resulted from pixel holes on or not. Confidence maps [ 19 ] of depth image were produced by combining mask images [ 21 ] ( ) with relative coarse depth images ( and were denoted as according to resolution.…”

“…As is shown in Figure 3 , pixel holes in mask images ( ) suggest that there are lots of missing pixels in ground truth depth images, which inevitably causes deviation to supervised learning. Therefore, we adopted a multi-layer convolution network ( ) for producing confidence map [ 19 ] of input depth images. stands for scale value of images, i.e., if the resolution of 2D images can be denoted as , then the corresponding is defined as .…”

“…Surface normal guidance has been introduced by previous studies [ 10 , 19 , 20 ], where they employed surface normal maps as 3D cues for improving the geometric quality of monocular depth images. Qi et al [ 10 ] jointly calculated depth and surface normal from a single image, making the final estimation geometrically more precise.…”

“…Qi et al [ 10 ] jointly calculated depth and surface normal from a single image, making the final estimation geometrically more precise. In work of Zeng et al [ 19 ], a skip-connected architecture was proposed to fuse features from different layers for surface normal estimation. A novel 3D geometric feature virtual normal was proposed by Yin et al [ 20 ] to refine the predicted depth maps.…”

“…Alhashim et al [ 6 ] proved that a very simple transfer learning-based decoder robustly achieves high-resolution depth maps. Previous studies [ 6 , 9 , 19 ] proved that the Dense-net is more suitable for depth estimation than models like SE-Net, Res-Net, and Mobile-net. However, experiments in references [ 6 , 9 ] showed that pre-trained deep learning structures based on Densenet-161, Densenet-169, and Densenet-201 models cannot afford real-time depth estimation.…”