Depth Super-Resolution From RGB-D Pairs With Transform and Spatial Domain Regularization

Abstract

This paper proposes a depth super-resolution method with both transform and spatial domain regularization. In the transform domain regularization, nonlocal correlations are exploited via an auto-regressive model, where each patch is further sparsified with a locally-trained transform to consider intra-patch correlations. In the spatial domain regularization, we propose a multi-directional total variation (MTV) prior to characterize the geometrical structures spatially orientated at arbitrary directions in depth maps. To achieve adaptive regularization, the MTV is weighted for each directional finite difference considering local characteristics of RGB-D data. We develop an accelerated proximal gradient algorithm to solve the proposed model. Quantitative and qualitative evaluations compared with state-of-the-art methods demonstrate that the proposed method achieves superior depth super-resolution performance for various configurations of magnification factors and datasets.

Deep edge map guided depth super resolution

“…The regularization-based methods utilize regularization terms to make the ill-posed depth SR problem well constrained. Common regularization includes nonlocal regularization [31,32], smoothness regularization [12,33], total variation (TV) regularization [34,13] and graph Laplacian regularization [35]. These regularizers greatly improve the depth SR performance.…”

“…To tackle this problem, there are many attempts [10,11] to mitigate negative effects of the color image, such as designing elaborate weighting factors [12,13,14], adopting joint guidance [15,16], learning complementary information of RGB-D pairs [17,18], and explicit inconsistency measurement [8,10]. Among these attempts, a simple and intuitive solution to avoid texture copy artifacts is utilizing mutual edges in an RGB-D pair as guidance [19,20].…”

“…What kind of SR subnetwork architecture to choose is also important. We adopt a progressive strategy in this paper, which has been proved to be effective in image SR [27,13]. Despite the same network architecture for each cascade level, previous works generally assign different network parameters for different levels.…”

“…We first warp the LR depth images to the view and size of the color images. Then we utilize the method in [13] to reconstruct the warped depth images and then refine them via manual adjustment to synthesize pseudo ground truth (GT). In summary, our dataset contains LR depth images, their corresponding high resolution (HR) color images, warped LR depth images, and the pseudo GT.…”

Perceptual Deep Depth Super-Resolution

“…Another strategy to tackle ambiguities in superresolution is to design sophisticated regularizers to balance the data-fidelity terms against a structural image prior [15,24,56]. In contrast to this approach, which requires custom hand-crafted regularized objectives and optimization procedures, we focus on the standard training strategy (i.e., gradient-based optimization of a CNN) while using a loss function that captures visual similarity.…”