3D Aware Correction and Completion of Depth Maps in Piecewise Planar Scenes

Abstract

Abstract. RGB-D sensors are popular in the computer vision community, especially for problems of scene understanding, semantic scene labeling, and segmentation. However, most of these methods depend on reliable input depth measurements. The reliability of these depth values deteriorates significantly with distance. In practice, unreliable depth measurements are discarded, thus, limiting the performance of methods that use RGB-D data. This paper studies how reliable depth values can be used to correct the unreliable ones, and how to complete (or extend) the available depth data beyond the raw measurements of the sensor (i.e. infer depth at pixels with unknown depth values), given a prior model on the 3D scene. We consider piecewise planar environments in this paper, since many indoor scenes with man-made objects can be modeled as such. We propose a framework that uses the RGB-D sensor's noise profile to adaptively and robustly fit plane segments (e.g. floor and ceiling) and iteratively complete the depth map, when possible. Depth completion is formulated as a discrete labeling problem (MRF) with hard constraints and solved efficiently using graph cuts. To regularize this problem, we exploit 3D and appearance cues that encourage pixels to take on depth values that will be compatible in 3D to the piecewise planar assumption. Extensive experiments, on a new large-scale and challenging dataset, show that our approach results in more accurate depth maps (with 20% more depth values) than those recorded by the RGB-D sensor. Additional experiments on the NYUv2 dataset show that our method generates more 3D aware depth. These generated depth maps can also be used to improve the performance of a state-of-the-art RGB-D SLAM method.

Deep Depth Completion of a Single RGB-D Image

“…Depth inpainting. Many methods have been proposed for filling holes in depth channels of RGB-D images, including ones that employ smoothness priors [30], fast marching methods [25,42], Navier-Stokes [6], anisotropic diffusion [41], background surface extrapolation [51,54,68], colordepth edge alignment [10,77,81], low-rank matrix completion [75], tensor voting [36], Mumford-Shah functional optimization [44], joint optimization with other properties of intrinsic images [4], and patch-based image synthesis [11,16,24]. Recently, methods have been proposed for inpainting color images with auto-encoders [70] and GAN architectures [58].…”

Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image

“…Image Inpainting Similar to geometry completion, researchers have employed various priors or optimized models to complete a depth image [38], [39], [40], [41], [42], [43], [44], [45]. The patch-based image synthesis idea is also applied [46], [47].…”

Deep Surface Normal Estimation With Hierarchical RGB-D Fusion

“…However, the depth camera used in common datasets, e.g., NYUv2 [22], Matterport3D [2], ScanNet [6] often fails to sense the depth on glossy, bright, transparent and faraway surfaces [32,29], resulting in holes and corruptions in the obtained depth images. To overcome missing pixels in normal map inferred from depth, some works proposed to inpaint depth images using RGB images [7,10,16,25,31]. Silberman et al [22] used optimization-based method [14] to fill the holes in depth maps.…”