Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures

Marin, Giulio; Zanuttigh, Pietro; Mattoccia, Stefano

doi:10.1007/978-3-319-46478-7_24

Cited by 42 publications

(73 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 7 reports experiment from sequence 2011 09 26 0011 of the KITTI raw dataset [6]. We compare our framework with fusion strategies proposed by Martins et al [18] and Marin et al [17], combining outputs by the stereo networks respectively with monocular estimates (using the network by Guo et al [9]) and Lidar, reporting the ideal result as in [17]. Ground-truth labels for evaluation are provided by [39].…”

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

“…<2% avg. All NoG All NoG iResNet [14] 18.42 18.37 1.28 1.28 iResNet+Martins et al [18] 18 is not limited to pixels with associated Lidar measurement in contrast to fusion techniques [17].…”

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

“…In this paper, we propose to leverage a small set of sparse depth measurements to obtain, with deep stereo networks, dense and accurate estimations in any environment. It is worth pointing out that our proposal is different from depth fusion strategies (e.g., [17,21,5,1]) aimed at combining the output of active sensors and stereo algorithms such as Semi-Global Matching [10]. Indeed, such methods mostly aim at selecting the most reliable depth measurements from the multiple available using appropriate frameworks whereas our proposal has an entirely different goal.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

93.21% 4.36% (a) (b) (c) Figure 1. Guided stereo matching. (a) Challenging, reference image from KITTI 2015 [20] and disparity maps estimated by (b) iResNet [14] trained on synthetic data [19], or (c) guided by sparse depth measurements (5% density). Error rate (> 3) superimposed on each map. AbstractStereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm leveraging a small amount of sparse, yet reliable depth measurements retrieved from an external source enabling to ameliorate this weakness. The additional sparse cues required by our method can be obtained with any strategy (e.g., a LiDAR) and used to enhance features linked to corresponding disparity hypotheses. Our formulation is general and fully differentiable, thus enabling to exploit the additional sparse inputs in pre-trained deep stereo networks as well as for training a new instance from scratch. Extensive experiments on three standard datasets and two stateof-the-art deep architectures show that even with a small set of sparse input cues, i) the proposed paradigm enables significant improvements to pre-trained networks. Moreover, ii) training from scratch notably increases accuracy and robustness to domain shifts. Finally, iii) it is suited and effective even with traditional stereo algorithms such as SGM.

show abstract

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

“…<2% avg. All NoG All NoG iResNet [14] 18.42 18.37 1.28 1.28 iResNet+Martins et al [18] 18 is not limited to pixels with associated Lidar measurement in contrast to fusion techniques [17].…”

Section: Experiments With Lidar Measurementsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Guided Stereo Matching

Poggi

Pallotti

Tosi

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In recent years, different kinds of depth fusion methods have emerged in different sub-tasks, such as stereo-ToF fusion ( [6,7,8]), stereo-stereo fusion ( [9]), Lidar-stereo fusion ( [10,11]) and general depth fusion ( [12]). Additionally, deep-learning based methods perform much better than the rest.…”

Section: Introductionmentioning

confidence: 99%

UDFNET: Unsupervised Disparity Fusion with Adversarial Networks

Fisher

2019

2019 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Existing disparity fusion methods based on deep learning achieve state-of-the-art performance, but they require ground truth disparity data to train. As far as I know, this is the first time an unsupervised disparity fusion not using ground truth disparity data has been proposed. In this paper, a mathematical model for disparity fusion is proposed to guide an adversarial network to train effectively without ground truth disparity data. The initial disparity maps are inputted from the left view along with auxiliary information (gradient, left & right intensity image) into the refiner and the refiner is trained to output the refined disparity map registered on the left view. The refined left disparity map and left intensity image are used to reconstruct a fake right intensity image. Finally, the fake and real right intensity images (from the right stereo vision camera) are fed into the discriminator. In the model, the refiner is trained to output a refined disparity value close to the weighted sum of the disparity inputs for global initialisation. Then, three refinement principles are adopted to refine the results further. (1) The reconstructed intensity error between the fake and real right intensity image is minimised.(2) The similarities between the fake and real right image in different receptive fields are maximised.(3) The refined disparity map is smoothed based on the corresponding intensity image. The adversarial networks' architectures are effective for the fusion task. The fusion time using the proposed network is small. The network can achieve 90 fps using Nvidia Geforce GTX 1080Ti on the Kitti2015 dataset when the input resolution is 1242 * 375 (Width * Height) without downsampling and cropping. The accuracy of this work is equal to (or better than) the state-of-the-art supervised methods.

show abstract

“…The challenges mainly lie in two parts: the first is how to get the real uncertainty distribution information from the real sensors. In the recent years, an increasing number of researchers have been investigating how to estimate the uncertainty of the acquired data for different sensors, such as the Kinect sensor [20], the time of flight sensor [7], the structure from motion sensor [10] and the stereo vision sensor [18]. These suggest using physical noise models for each point to represent their individual occurrence probability in 3D space.…”

Section: Introductionmentioning

confidence: 99%

DUGMA: Dynamic Uncertainty-Based Gaussian Mixture Alignment

Tyleček³

et al. 2018

2018 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

Accurately registering point clouds from a cheap lowresolution sensor is a challenging task. Existing rigid registration methods failed to use the physical 3D uncertainty distribution of each point from a real sensor in the dynamic alignment process. It is mainly because the uncertainty model for a point is static and invariant and it is hard to describe the change of these physical uncertainty models in different views. Additionally, the existing Gaussian mixture alignment architecture cannot efficiently implement these dynamic changes.This paper proposes a simple architecture combining error estimation from sample covariances and dynamic global probability alignment using the convolution of uncertaintybased Gaussian Mixture Models (GMM). Firstly, we propose an efficient way to describe the change of each 3D uncertainty model, which represents the structure of the point cloud better. Unlike the invariant GMM (representing a fixed point cloud) in traditional Gaussian mixture alignment, we use two uncertainty-based GMMs that change and interact with each other in each iteration. In order to have a wider basin of convergence than other local algorithms, we design a more robust energy function by convolving efficiently the two GMMs over the whole 3D space.Tens of thousands of trials have been conducted on hundreds of models from multiple datasets to demonstrate the proposed method's superior performance compared with the current state-of-the-art methods. All the materials including our code are available from https://github. com/Canpu999/DUGMA.

show abstract

Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures

Cited by 42 publications

References 27 publications

Guided Stereo Matching

Guided Stereo Matching

UDFNET: Unsupervised Disparity Fusion with Adversarial Networks

DUGMA: Dynamic Uncertainty-Based Gaussian Mixture Alignment

Contact Info

Product

Resources

About