Perceptual Deep Depth Super-Resolution

Abstract

RGBD images, combining high-resolution color and lower-resolution depth from various types of depth sensors, are increasingly common. One can significantly improve the resolution of depth maps by taking advantage of color information; deep learning methods make combining color and depth information particularly easy.However, fusing these two sources of data may lead to a variety of artifacts. If depth maps are used to reconstruct 3D shapes, e.g., for virtual reality applications, the visual quality of upsampled images is particularly important.The main idea of our approach is to measure the quality of depth map upsampling using renderings of resulting 3D surfaces. We demonstrate that a simple visual appearancebased loss, when used with either a trained CNN or simply a deep prior, yields significantly improved 3D shapes, as measured by a number of existing perceptual metrics. We compare this approach with a number of existing optimization and learning-based techniques.

Deep edge map guided depth super resolution

“…The applications of convolutional neural networks (CNNs) greatly improve depth SR performance benefiting from advanced network architectures [37,38], effective loss functions [39,40] and massive data. In CNN-based methods, color image is a useful extra information to increase the reconstruction accuracy [41].…”

Color-Guided Depth Map Super-Resolution Using a Dual-Branch Multi-Scale Residual Network with Channel Interaction

“…These DCNNs adopt a residual network or multi-scale upsampling mechanism like our proposed network but the ways in which the intensity image guides the process are different, which determines a difference in the severity of artifacts. Voynov et al [26] tried to avoid artifacts for virtual reality applications and they measured the quality of a depth map upsampling using renderings of the resulting 3D surfaces.…”

“…To test the generalization of our proposed network, we selected three RGBD images from different databases to form a new dataset Mixture in which image Lucy from the SimGeo dataset [26], image Plant from the ICL-NUIM (Imperial College London-National University of Ireland Maynooth) dataset [38], and image Vintage from Middlebury dataset were considered. The model we used for evaluation was the same as the model tested on datasets A, B, and C without fine-tuning, and the evaluation criterion was still the RMSE.…”

“…The model we used for evaluation was the same as the model tested on datasets A, B, and C without fine-tuning, and the evaluation criterion was still the RMSE. We mainly tested our method at the upscaling factors of 4 and 8, in comparison with methods from References [23,26,[39][40][41]. Our method produced the best performance on the image from the Middlebury dataset and performed nearly 20% better than the sub-optimal result (see Table 4).…”

Learning to Approximate Directional Fields Defined Over 2D Planes

“…With the guidance of an appropriately designed directional field, both topological (e.g., placement of singularity points) and geometric (e.g., smoothness) properties of the underlying geometric structure may be efficiently derived. Other applications which could benefit from learnable directional fields include remote sensing [11,3,14,9], RGBD data processing [19] and related applications [1,5], shape retrieval [13]. However, obtaining a robust approximation of a directional field from raw input data is a challenging problem in many instances.…”