Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation

Abstract

In this work we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or Time of Flight cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primaldual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.

Building Scene Models by Completing and Hallucinating Depth and Semantics

“…For the visible layer, depth estimation translates to the usual depth completion problem. We therefore compare the results of our visible layer with the of the classical method of [29], and with the more recent technique of [27].…”

“…We therefore apply the technique of [28] to inpaint this area, which, to the best of our knowledge, remains the most mature method when it comes to depth completion without intensity information. This yields two baselines, which we will refer to as Baseline-1 (semantic segmentation followed by [29] + [28]) and Baseline-2 (semantic segmentation followed by [27] + [28]). To compare the different algorithms, we make use of the following metrics:1) visible-rmse: the-root-mean-square-error (rmse) for the entire depth map; 2)hidden-rmse: the rmse for the depth map hallucinated underneath the ground truth foreground mask.…”

“…Herrera et al [26] propose an MRF with second-order prior to inpaint piecewise planar depth maps. In [27], depth completion is formulated within a total variation framework where image cues guide the completion process. A different approach to depth completion consists of treating a depth map as an intensity image, and rely on standard image inpainting algorithms, such as [28] and [29].…”

“…Here, we further rely on the oriented gradient operator K of [27], which computes an image-adaptive gradient for each channel of u v and s v . More specifically, the oriented gradient operator K at location x is defined by T I (x)∇, where T I is an image-based anisotropic diffusion tensor.…”

Reconstruction of high-resolution Depth Map using Sparse Linear Model

“…HQDMU takes multiple factors to weight nodes on depth edges and takes similarities of textures in local regions to weight nodes on other parts of depth maps. In order to generate better depth edges, Ferstl et al [13] used a second-order TGV(Total Generative Variation) of color textures to segment the color image into slices. Then they reconstructed depth maps by solving a convex optimization problem.…”