Frequency-Dependent Depth Map Enhancement via Iterative Depth-Guided Affine Transformation and Intensity-Guided Refinement

Zuo, Yifan; Fang, Yuming; An, Ping; Shang, Xiwu; Yang, Junnan

doi:10.1109/tmm.2020.2987706

Cited by 22 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Zhao et al [152] propose a color-depth conditional generative network (CDcGAN) to simultaneously super-resolve the low-resolution depth and color images. To reduce the artifacts caused by the differences of distributions between the depth and its corresponding color image, Zuo et al [159] propose a depth-guided affine transformation network in which depth-guided color feature filtering and color-guided depth feature refinement are performed iteratively to progressively enhance the network representation ability. Moreover, all the refined depth features are concatenated to make full use of the iterations.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Guided Depth Map Super-resolution: A Survey

Zhang¹,

Liu²,

Jiang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Guided depth map super-resolution (GDSR), which aims to reconstruct a high-resolution (HR) depth map from a low-resolution (LR) observation with the help of a paired HR color image, is a longstanding and fundamental problem, it has attracted considerable attention from computer vision and image processing communities. A myriad of novel and effective approaches have been proposed recently, especially with powerful deep learning techniques. This survey is an effort to present a comprehensive survey of recent progress in GDSR. We start by summarizing the problem of GDSR and explaining why it is challenging. Next, we introduce some commonly used datasets and image quality assessment methods. In addition, we roughly classify existing GDSR methods into three categories, i.e., filtering-based methods, prior-based methods, and learning-based methods. In each category, we introduce the general description of the published algorithms and design principles, summarize the representative methods, and discuss their highlights and limitations. Moreover, the depth related applications are introduced. Furthermore, we conduct experiments to evaluate the performance of some representative methods based on unified experimental configurations, so as to offer a systematic and fair performance evaluation to readers. Finally, we conclude this survey with possible directions and open problems for further research. All the related materials can be found at https://github.com/zhwzhong/Guided-Depth-Map-Super-resolution-A-Survey.CCS Concepts: • General and reference → Surveys and overviews.

show abstract

Section: Deep Learning Methodsmentioning

confidence: 99%

“…CGN [159] TMM-2020 Proposes a coarse to fine framework which contains a depth-guided intensity features filtering module and a intensity-guided depth features refinement module to mitigate the artifacts caused by edge misalignment.…”

Section: Dsrn [127] Pr-2020mentioning

confidence: 99%

Guided Depth Map Super-resolution: A Survey

Zhang¹,

Liu²,

Jiang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To address texture-copy artifacts, Zuo et al [44] proposed a local affine transformation to filter out the unrelated intensity features explicitly by Hadamard product operations. Deng et al [45] split the common information from different modalities by designing extraction modules for unique feature and common feature respectively.…”

Section: Deep Depth Image Super Resolutionmentioning

confidence: 99%

Deep edge map guided depth super resolution

Jiang

Yue

Lai

et al. 2021

Signal Processing: Image Communication

View full text Add to dashboard Cite

Accurate edge reconstruction is critical for depth map super resolution (SR). Therefore, many traditional SR methods utilize edge maps to guide depth SR. However, it is difficult to predict accurate edge maps from low resolution (L-R) depth maps. In this paper, we propose a deep edge map guided depth SR method, which includes an edge prediction subnetwork and an SR subnetwork. The edge prediction subnetwork takes advantage of the hierarchical representation of color and depth images to produce accurate edge maps, which promote the performance of SR subnetwork. The SR subnetwork is a disentangling cascaded network to progressively upsample SR result, where every level is made up of a weight sharing module and an adaptive module. The weight sharing module extracts the general features in different levels, while the adaptive module transfers the general features to the specific features to adapt to different degraded inputs. Quantitative and qualitative evaluations on various datasets with different magnification factors demonstrate the effectiveness and promising performance of the proposed method. In addition, we construct a benchmark dataset captured by Kinect-v2 to facilitate research on real-world depth map SR.

show abstract

Section: Introductionmentioning

confidence: 99%

“…Li et al [8] introduced synchronized RGB images to align with the depth image, and preprocessed the color image and the depth image to extract the effective supporting edge area, which ensured the edge details of the image while repairing the depth map. Zuo et al [9] proposed a frequency-dependent depth map enhancement algorithm via iterative depth guided affine transformation and intensity-guided refinement, and improved performances based on qualitative and quantitative evaluations are demonstrated. To approximate the real depth map, it is hard to train the depth map to be repaired directly, however, if the jump structure of residual network (ResNet) is adopted, then the training goal becomes to approximate the difference, which is between the depth map to be repaired and the real depth map.…”

Section: Introductionmentioning

confidence: 99%

A Residual Network and FPGA Based Real-Time Depth Map Enhancement System

Sun

Gao

et al. 2021

Entropy

View full text Add to dashboard Cite

Depth maps obtained through sensors are often unsatisfactory because of their low-resolution and noise interference. In this paper, we propose a real-time depth map enhancement system based on a residual network which uses dual channels to process depth maps and intensity maps respectively and cancels the preprocessing process, and the algorithm proposed can achieve real-time processing speed at more than 30 fps. Furthermore, the FPGA design and implementation for depth sensing is also introduced. In this FPGA design, intensity image and depth image are captured by the dual-camera synchronous acquisition system as the input of neural network. Experiments on various depth map restoration shows our algorithms has better performance than existing LRMC, DE-CNN and DDTF algorithms on standard datasets and has a better depth map super-resolution, and our FPGA completed the test of the system to ensure that the data throughput of the USB 3.0 interface of the acquisition system is stable at 226 Mbps, and support dual-camera to work at full speed, that is, 54 fps@ (1280 × 960 + 328 × 248 × 3).

show abstract

Frequency-Dependent Depth Map Enhancement via Iterative Depth-Guided Affine Transformation and Intensity-Guided Refinement

Cited by 22 publications

References 44 publications

Guided Depth Map Super-resolution: A Survey

Guided Depth Map Super-resolution: A Survey

Deep edge map guided depth super resolution

A Residual Network and FPGA Based Real-Time Depth Map Enhancement System

Contact Info

Product

Resources

About