Deep Learning for Confidence Information in Stereo and ToF Data Fusion

Agresti, Gianluca; Minto, Ludovico; Marin, Giulio; Zanuttigh, Pietro

doi:10.1109/iccvw.2017.88

Cited by 36 publications

(66 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we propose to leverage a small set of sparse depth measurements to obtain, with deep stereo networks, dense and accurate estimations in any environment. It is worth pointing out that our proposal is different from depth fusion strategies (e.g., [17,21,5,1]) aimed at combining the output of active sensors and stereo algorithms such as Semi-Global Matching [10]. Indeed, such methods mostly aim at selecting the most reliable depth measurements from the multiple available using appropriate frameworks whereas our proposal has an entirely different goal.…”

Section: Introductionmentioning

confidence: 99%

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

93.21% 4.36% (a) (b) (c) Figure 1. Guided stereo matching. (a) Challenging, reference image from KITTI 2015 [20] and disparity maps estimated by (b) iResNet [14] trained on synthetic data [19], or (c) guided by sparse depth measurements (5% density). Error rate (> 3) superimposed on each map. AbstractStereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm leveraging a small amount of sparse, yet reliable depth measurements retrieved from an external source enabling to ameliorate this weakness. The additional sparse cues required by our method can be obtained with any strategy (e.g., a LiDAR) and used to enhance features linked to corresponding disparity hypotheses. Our formulation is general and fully differentiable, thus enabling to exploit the additional sparse inputs in pre-trained deep stereo networks as well as for training a new instance from scratch. Extensive experiments on three standard datasets and two stateof-the-art deep architectures show that even with a small set of sparse input cues, i) the proposed paradigm enables significant improvements to pre-trained networks. Moreover, ii) training from scratch notably increases accuracy and robustness to domain shifts. Finally, iii) it is suited and effective even with traditional stereo algorithms such as SGM.

show abstract

Section: Introductionmentioning

confidence: 99%

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…[38] and the confidence score c ∈ [0, 1] that locally ties the solution u to the initial estimateû. To compute u we use a CNN that is split into different parts to deliver the inputs for our optimization stage solving (1) or (2). In particular, we perform dense pixel-wise matching using network generated features, refine the matches by locally fitting a quadratic to the cost and employ the arg min cost solution as initial estimate.…”

Section: Methodsmentioning

confidence: 99%

“…In the context of stereo matching confidence estimation is a well studied problem [16]. Lately also CNN based solutions have been proposed [2,26]. Compared to the general problem, our solution is directly task related.…”

Section: Introduction and Related Workmentioning

confidence: 99%

Learning Energy Based Inpainting for Optical Flow

Vogel

Knöbelreiter

Pock

2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

Modern optical flow methods are often composed of a cascade of many independent steps or formulated as a black box neural network that is hard to interpret and analyze. In this work we seek for a plain, interpretable, but learnable solution. We propose a novel inpainting based algorithm that approaches the problem in three steps: feature selection and matching, selection of supporting points and energy based inpainting. To facilitate the inference we propose an optimization layer that allows to backpropagate through 10K iterations of a first-order method without any numerical or memory problems. Compared to recent state-of-the-art networks, our modular CNN is very lightweight and competitive with other, more involved, inpainting based methods.

show abstract

“…In summary, our contributions are as follows: (1) we demonstrate that a simple and efficient visual differencebased metric for depth map comparison can be, on the one hand, easily combined with neural network-based wholeimage upsampling techniques, and, on the other hand, is correlated with established proxies for human perception, validated with respect to experimental measurements; (2) we demonstrate with extensive comparisons that with the use of this metric two methods of depth map superresolution, one based on a trainable CNN and the other based on the deep prior, yield high-quality results as measured by multiple perceptual metrics. To the best of our knowledge, our paper is the first to systematically study the performance of visual difference-based depth superresolution across a variety of datasets, methods, and quality measures, including a basic human evaluation.…”

Section: Introductionmentioning

confidence: 99%

Perceptual Deep Depth Super-Resolution

Voynov

Artemov

Egiazarian

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

RGBD images, combining high-resolution color and lower-resolution depth from various types of depth sensors, are increasingly common. One can significantly improve the resolution of depth maps by taking advantage of color information; deep learning methods make combining color and depth information particularly easy.However, fusing these two sources of data may lead to a variety of artifacts. If depth maps are used to reconstruct 3D shapes, e.g., for virtual reality applications, the visual quality of upsampled images is particularly important.The main idea of our approach is to measure the quality of depth map upsampling using renderings of resulting 3D surfaces. We demonstrate that a simple visual appearancebased loss, when used with either a trained CNN or simply a deep prior, yields significantly improved 3D shapes, as measured by a number of existing perceptual metrics. We compare this approach with a number of existing optimization and learning-based techniques.

show abstract

Deep Learning for Confidence Information in Stereo and ToF Data Fusion

Cited by 36 publications

References 28 publications

Guided Stereo Matching

Guided Stereo Matching

Learning Energy Based Inpainting for Optical Flow

Perceptual Deep Depth Super-Resolution

Contact Info

Product

Resources

About