Explicit Edge Inconsistency Evaluation Model for Color-Guided Depth Map Enhancement

Zuo, Yifan; Wu, Qiang; Zhang, Jian; An, Ping

doi:10.1109/tcsvt.2016.2609438

Cited by 51 publications

(26 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ferstl et al [14] used a secondary generalized variable guided by an anisotropic diffusion tensor extracted from an HR color image to limit a regularized HR depth map. Zuo et al [15,16] measured the discontinuities of edges between a color image and a depth map in an MRF, and these discontinuities can be reflected in the edge weight of the minimum spanning tree. Yang et al [17] proposed a novel depth map SR method guided by a color image by using an auto-regression model.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Color-Guided Depth Map Super-Resolution Using a Dual-Branch Multi-Scale Residual Network with Channel Interaction

Chen

Gao

2020

Sensors

View full text Add to dashboard Cite

We designed an end-to-end dual-branch residual network architecture that inputs a low-resolution (LR) depth map and a corresponding high-resolution (HR) color image separately into the two branches, and outputs an HR depth map through a multi-scale, channel-wise feature extraction, interaction, and upsampling. Each branch of this network contains several residual levels at different scales, and each level comprises multiple residual groups composed of several residual blocks. A short-skip connection in every residual block and a long-skip connection in each residual group or level allow for low-frequency information to be bypassed while the main network focuses on learning high-frequency information. High-frequency information learned by each residual block in the color image branch is input into the corresponding residual block in the depth map branch, and this kind of channel-wise feature supplement and fusion can not only help the depth map branch to alleviate blur in details like edges, but also introduce some depth artifacts to feature maps. To avoid the above introduced artifacts, the channel interaction fuses the feature maps using weights referring to the channel attention mechanism. The parallel multi-scale network architecture with channel interaction for feature guidance is the main contribution of our work and experiments show that our proposed method had a better performance in terms of accuracy compared with other methods.

show abstract

Section: Related Workmentioning

confidence: 99%

“…PixelShuffle (B) x,y,c = B x/r,y/r,C•r•mod(y,r)+C•mod(x,r)+c (15) where x and y are the output pixel coordinates of the cth feature map in HR space. The feature maps from the LR space are built into HR feature maps through the pixel-shuffling layer.…”

Section: Rgb Image Network Branchmentioning

confidence: 99%

Color-Guided Depth Map Super-Resolution Using a Dual-Branch Multi-Scale Residual Network with Channel Interaction

Chen

Gao

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Li et al [29] proposed a hierarchical global optimization framework based on weighted least squares (WLS) technique. In the MRF framework, Zuo et al [30] explicitly model the discontinuity inconsistency between the depth map and color image in the smoothness term to reduce texture copy and blurring artifacts.…”

Section: A Depth Image Recoverymentioning

confidence: 99%

Global Auto-Regressive Depth Recovery via Iterative Non-Local Filtering

Yang

Frossard

2019

IEEE Trans. on Broadcast.

View full text Add to dashboard Cite

Existing depth sensing techniques have many shortcomings in terms of resolution, completeness, and accuracy. The performance of 3-D broadcasting systems is therefore limited by the challenges of capturing high-resolution depth data. In this paper, we present a novel framework for obtaining high-quality depth images and multi-view depth videos from simple acquisition systems. We first propose a single depth image recovery algorithm based on auto-regressive (AR) correlations. A fixedpoint iteration algorithm under the global AR modeling is derived to efficiently solve the large-scale quadratic programming. Each iteration is equivalent to a nonlocal filtering process with a residue feedback. Then, we extend our framework to an ARbased multi-view depth video recovery framework, where each depth map is recovered from low-quality measurements with the help of the corresponding color image, depth maps from neighboring views, and depth maps of temporally adjacent frames. AR coefficients on nonlocal spatiotemporal neighborhoods in the algorithm are designed to improve the recovery performance. We further discuss the connections between our model and other methods like graph-based tools, and demonstrate that our algorithms enjoy the advantages of both global and local methods. Experimental results on both the Middleburry datasets and other captured datasets finally show that our method is able to improve the performances of depth images and multi-view depth videos recovery compared with state-of-the-art approaches. Index Terms-Depth recovery, multi-view depth video, autoregressive model, nonlocal correlation, iterative filtering. I. INTRODUCTION 3 D IMAGING applications have rapidly gained interest in recent years, as they promise to enhance the visual experience by extending the conventional 2D video to a more immersive experience. However, the technologies constituting 3D imaging systems are not fully mature yet to

show abstract

“…Therefore, they cannot adaptively control the efforts of the guidance from the HR color image when enhancing the LR depth map. Our previous work 9 for the first time explicitly calculates the inconsistency between the depth edges and the corresponding color edges through global optimization. Although the performance of this type of method is significantly improved, the improvement is limited because only the handcraft features (e.g., edge, segment) are used.…”

Section: Introductionmentioning

confidence: 99%

“…Based on the analysis above, this paper proposes an approach that combines model-based prior (i.e., a more accuracy model to represent the property of edge cooccurrence on color-depth image pair) with the data-driven one to better mitigate texture-copying artifacts and restore more details for upsampling the LR depth map. Compared with our previous work, 9 the main contribution of this article is to consider model-based and data-driven priors as regularizations on the depth map upsampling. Such two parts complement each other, which are not considered in Ref.…”

Section: Introductionmentioning

confidence: 99%

Integrated cosparse analysis model with explicit edge inconsistency measurement for guided depth map upsampling

Zuo

et al. 2018

J. Electron. Imag.

Self Cite

View full text Add to dashboard Cite

A low-resolution depth map can be upsampled through the guidance from the registered highresolution color image. This type of method is so-called guided depth map upsampling. Among the existing methods based on Markov random field (MRF), either data-driven or model-based prior is adopted to construct the regularization term. The data-driven prior can implicitly reveal the relation between color-depth image pair by training on external data. The model-based prior provides the anisotropic smoothness constraint guided by highresolution color image. These types of priors can complement each other to solve the ambiguity in guided depth map upsampling. An MRF-based approach is proposed that takes both of them into account to regularize the depth map. Based on analysis sparse coding, the data-driven prior is defined by joint cosparsity on the vectors transformed from color-depth patches using the pair of learned operators. It is based on the assumption that the cosupports of such bimodal image structures computed by the operators are aligned. The edge inconsistency measurement is explicitly calculated, which is embedded into the model-based prior. It can significantly mitigate texture-copying artifacts. The experimental results on Middlebury datasets demonstrate the validity of the proposed method that outperforms seven state-of-the-art approaches.

show abstract

Explicit Edge Inconsistency Evaluation Model for Color-Guided Depth Map Enhancement

Cited by 51 publications

References 55 publications

Color-Guided Depth Map Super-Resolution Using a Dual-Branch Multi-Scale Residual Network with Channel Interaction

Color-Guided Depth Map Super-Resolution Using a Dual-Branch Multi-Scale Residual Network with Channel Interaction

Global Auto-Regressive Depth Recovery via Iterative Non-Local Filtering

Integrated cosparse analysis model with explicit edge inconsistency measurement for guided depth map upsampling

Contact Info

Product

Resources

About