Patch Based Synthesis for Single Depth Image Super-Resolution

Abstract

Abstract. We present an algorithm to synthetically increase the resolution of a solitary depth image using only a generic database of local patches. Modern range sensors measure depths with non-Gaussian noise and at lower starting resolutions than typical visible-light cameras. While patch based approaches for upsampling intensity images continue to improve, this is the first exploration of patching for depth images. We match against the height field of each low resolution input depth patch, and search our database for a list of appropriate high resolution candidate patches. Selecting the right candidate at each location in the depth image is then posed as a Markov random field labeling problem. Our experiments also show how important further depth-specific processing, such as noise removal and correct patch normalization, dramatically improves our results. Perhaps surprisingly, even better results are achieved on a variety of real test scenes by providing our algorithm with only synthetic training depth data.

Depth map inpainting and super-resolution based on internal statistics of geometry and appearance

“…Second, we compared our method with three state-ofthe-art SR algorithms: sparse coding based image superresolution (ScSR) [22], patch-based synthesis (PbsSR) [14], and weighted mode filtering (WMF) [9], and also compared it with nearest neighbor interpolation (NN). We also applied our method without the texture/sparsity constraints to confirm their contributions to our method.…”

“…3. We show that our method WMF [9] PbsSR [14] Input and ground truth depth map Registered Color Image outperforms all of the state-of-the-art algorithms on the art and mebius datasets and shows the 2nd best performance on the fountain and herzjesu datasets while ScSR suffers from blurring effects, WMF appears to be sensitive to texture edges and PbsSR has problems around the object boundaries.…”

“…More recently, learning-based single image super-resolution techniques were integrated into depth-map super-resolution to handle dynamic scenes. Aodha et al [14] learned the statistics of low-resolution and high-resolution depth patches from external training sets and used them to upsample a new low-resolution depth map in the MRF formation. In contrast, Li et al [15] derived a mapping function from a low-resolution depth patch and its corresponding color patch to a high-resolution depth patch via sparse coding.…”

Building Scene Models by Completing and Hallucinating Depth and Semantics

“…This is typically achieved via Markov Random Fields [6,21,7], bilateral filtering [22], layered representations [23], patch-based approaches [10,11], or depth transfer [8,9]. These approaches, however, inherently assume to have access to regularly-spaced depth measurements, and thus cannot handle large holes in depth maps.…”

“…For instance, depth super-resolution [6][7][8][9][10][11] tackles the sparseness issue and attempts to densify the observed depth data. Typically, however, existing methods assume that the measurements are regularly spaced, and are thus ill-suited to handle large holes.…”