A Deep Primal-Dual Network for Guided Depth Super-Resolution

Riegler, G. R.; Ferstl, David; Rüther, Matthias; Bischof, Horst

doi:10.5244/c.30.7

Cited by 72 publications

(61 citation statements)

References 29 publications

(63 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The total variations were expressed by layers with fixed parameters. Besides, Riegler et al [14] also proposed a novel DCNN based method by combining a DCNN with a non-local variational method. The corresponding HR color images were also utilized in the method.…”

Section: Depth Image Super-resolutionmentioning

confidence: 99%

“…To tackle this highly under-constrained problem, various methods have been proposed by exploiting the availability of large-scale training datasets. Even though deep convolutional neural network (DCNN) based methods have achieved great success in various vision tasks such as image deblurring [5] [6], image denoising [7] [8], monocular depth estimation [46], [56], saliency prediction [27], [28], and even color image superresolution (CSR) [9] [10] [11] [12], it is only very recently that the success of DCNN in color image super-resolution [9] [11] [12] has been extended to the task of depth map super-resolution [2] [13] [14] [15]. This is mainly due to the intrinsic differences between color images and depth maps, where the depth maps generally contain less textures and more sharp boundaries, and are usually degraded by noise due to the imprecise consumer depth cameras.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

Song¹,

Dai

Qin

2019

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Deep convolutional neural network (DCNN) has been successfully applied to depth map super-resolution and outperforms existing methods by a wide margin. However, there still exist two major issues with these DCNN based depth map super-resolution methods that hinder the performance: i) The low-resolution depth maps either need to be up-sampled before feeding into the network or substantial deconvolution has to be used; and ii) The supervision (high-resolution depth maps) is only applied at the end of the network, thus it is difficult to handle large up-sampling factors, such as ×8, ×16. In this paper, we propose a new framework to tackle the above problems. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. The novel view synthesis sub-task aims at generating (synthesizing) a depth map from different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, a multi-scale fusion strategy is proposed to effectively exploit the feature maps at different scales and handle the blocking effect. In this way, our proposed framework could deal with challenging depth map super-resolution efficiently under large up-sampling factors (e.g.×8, ×16). Our method only uses the low-resolution depth map as input, and the support of color image is not needed, which greatly reduces the restriction of our method. Extensive experiments on various benchmarking datasets demonstrate the superiority of our method over current state-of-the-art depth map super-resolution methods.

show abstract

Section: Depth Image Super-resolutionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

Song¹,

Dai

Qin

2019

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…To show the robustness of our models on noisy data, we train our models with the NYU v2 (Silberman et al 2012) dataset, and test them on noisy low-resolution depth images. We follow the protocol of (Riegler et al 2016a) to simulate noisy-low resolution depth images. The other experimental settings are the same as in Section 4.1.1.…”

Section: Resultsmentioning

confidence: 99%

3D Interpreter Networks for Viewer-Centered Wireframe Modeling

et al. 2018

View full text Add to dashboard Cite

Joint image filters are used to transfer structural details from a guidance picture used as a prior to a target image, in tasks such as enhancing spatial resolution and suppressing noise. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details and regress the filtering result. In this paper, we instead learn explicitly sparse and spatially-variant kernels. We propose a CNN architecture and its efficient implementation, called the deformable kernel network (DKN), that outputs sets of neighbors and the corresponding weights adaptively for each pixel. The filtering result is then computed as a weighted average. We also propose a fast version of DKN that runs about seventeen times faster for an image of size 640 × 480. We demonstrate the effectiveness and flexibility of our models on the tasks of depth map upsampling, saliency map upsampling, cross-modality image restoration, texture removal, and semantic segmentation. In particular, we show that the weighted averaging process with sparsely sampled 3 × 3 kernels outperforms the state of the art by a significant margin in all cases.

show abstract

“…However, we wanted to extend this idea further in text images, where the label propagation between neighbors should be encouraged but also restricted along the edges as shown in Fig.2(b). Use of PDUpdate scheme in tasks that involve total-variation formulation of energy function is already explored in depth super resolution Riegler et al (2016), and multi-class labeling problem Ochs et al (2015). We extend these ideas into a more stable architecture that permits end-to-end training within the intended theoretical framework of the underlying proximal operator, eliminating the exploding gradient problem due to its recurrent structure.…”

Section: Methodsmentioning

confidence: 99%

PDNet: Semantic segmentation integrated with a primal-dual network for document binarization

Ayyalasomayajula

Malmberg

Brun

2019

Pattern Recognition Letters

View full text Add to dashboard Cite

Binarization of digital documents is the task of classifying each pixel in an image of the document as belonging to the background (parchment/paper) or foreground (text/ink). Historical documents are often subjected to degradations, that make the task challenging. In the current work a deep neural network architecture is proposed that combines a fully convolutional network with an unrolled primal-dual network that can be trained end-to-end to achieve state of the art binarization on four out of seven datasets. Document binarization is formulated as an energy minimization problem. A fully convolutional neural network is trained for semantic segmentation of pixels that provides labeling cost associated with each pixel. This cost estimate is refined along the edges to compensate for any over or under estimation of the foreground class using a primal-dual approach. We provide necessary overview on proximal operator that facilitates theoretical underpinning required to train a primal-dual network using a gradient descent algorithm. Numerical instabilities encountered due to the recurrent nature of primal-dual approach are handled. We provide experimental results on document binarization competition dataset along with network changes and hyperparameter tuning required for stability and performance of the network. The network when pre-trained on synthetic dataset performs better as per the competition metrics.Accepted in PRL

show abstract

A Deep Primal-Dual Network for Guided Depth Super-Resolution

Cited by 72 publications

References 29 publications

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

3D Interpreter Networks for Viewer-Centered Wireframe Modeling

PDNet: Semantic segmentation integrated with a primal-dual network for document binarization

Contact Info

Product

Resources

About