Robust, Efficient Depth Reconstruction With Hierarchical Confidence-Based Matching

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

2021

Abstract. In recent years, the ability to assess the uncertainty of depth estimates in the context of dense stereo matching has received increased attention due to its potential to detect erroneous estimates. Especially, the introduction of deep learning approaches greatly improved general performance, with feature extraction from multiple modalities proving to be highly advantageous due to the unique and different characteristics of each modality. However, most work in the literature focuses on using only mono- or bi- or rarely tri-modal input, not considering the potential effectiveness of modalities, going beyond tri-modality. To further advance the idea of combining different types of features for confidence estimation, in this work, a CNN-based approach is proposed, exploiting uncertainty cues from up to four modalities. For this purpose, a state-of-the-art local-global approach is used as baseline and extended accordingly. Additionally, a novel disparity-based modality named warped difference is presented to support uncertainty estimation at common failure cases of dense stereo matching. The general validity and improved performance of the proposed approach is demonstrated and compared against the bi-modal baseline in an evaluation on three datasets using two common dense stereo matching techniques.

Section: Related Workmentioning

confidence: 99%

Learning Multi-Modal Features for Dense Matching-Based Confidence Estimation

Heinrich

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

2021

“…A good overview of the commonly used hand-crafted metrics is given in (Hu and Mordohai, 2012). Similar to other computer vision fields, more and more approaches based on deep learning (Poggi and Mattoccia, 2016;Mehltretter and Heipke, 2019;Kendall and Gal, 2017) and other machine learning techniques (Sun et al, 2017;Batsos et al, 2018) have been proposed in the literature. While a majority of these deep learning-based uncertainty estimation methods, operate on extracted patches from disparity maps only (Poggi and Mattoccia, 2016) or additionally take the RGB reference image into account (Fu et al, 2019), Mehltretter and Heipke (2019) utilise the information contained in the 3D cost volume.…”

Section: Related Workmentioning

confidence: 99%

Mixed Probability Models for Aleatoric Uncertainty Estimation in the Context of Dense Stereo Matching

Zhang

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

2021

Abstract. The ability to identify erroneous depth estimates is of fundamental interest. Information regarding the aleatoric uncertainty of depth estimates can be, for example, used to support the process of depth reconstruction itself. Consequently, various methods for the estimation of aleatoric uncertainty in the context of dense stereo matching have been presented in recent years, with deep learning-based approaches being particularly popular. Among these deep learning-based methods, probabilistic strategies are increasingly attracting interest, because the estimated uncertainty can be quantified in pixels or in metric units due to the consideration of real error distributions. However, existing probabilistic methods usually assume a unimodal distribution to describe the error distribution while simply neglecting cases in real-world scenarios that could violate this assumption. To overcome this limitation, we propose two novel mixed probability models consisting of Laplacian and Uniform distributions for the task of aleatoric uncertainty estimation. In this way, we explicitly address commonly challenging regions in the context of dense stereo matching and outlier measurements, respectively. To allow a fair comparison, we adapt a common neural network architecture to investigate the effects of the different uncertainty models. In an extensive evaluation using two datasets and two common dense stereo matching methods, the proposed methods demonstrate state-of-the-art accuracy.

“…The approaches of the second group combine certain of those features to form more accurate and robust measures. Beside linear aggregation [6], random forest based combinations are especially popular [3,16,7,10,2]. The transition to the third group is accomplished by utilising neuronal networks to carry out the combination task [17,14].…”

Section: Confidence Estimationmentioning

confidence: 99%

“…This allows to filter out local outliers from disparity maps and thus to subsequently adjust the ratio of density and reliability. The proposed applications are diverse: Confidence maps are used as weighting-schemes to combine multiple stereo matching algorithms [1,2], different cost functions [3,4,5] or to fuse cost volumes [6] for multi-view stereo, in a reasonable way. Confidence maps are furthermore used to improve the process of depth reconstruction itself: They allow to modulate cost functions in order to adjust the influence of a specific disparity assignment on its neighbours during optimisation [7,8].…”

Section: Introductionmentioning

confidence: 99%

“…Confidence maps are furthermore used to improve the process of depth reconstruction itself: They allow to modulate cost functions in order to adjust the influence of a specific disparity assignment on its neighbours during optimisation [7,8]. Finally, also Semiglobal Matching (SGM) [9] can be improved by introducing a confidence-based adjustment of the penalties [6] or by applying a weighted aggregation scheme for combining different paths within the optimisation process [10]. Figure 1: Confidence estimation on an image of KITTI 2015 dataset [11].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CNN-Based Cost Volume Analysis as Confidence Measure for Dense Matching

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Heipke

2019

Due to its capability to identify erroneous disparity assignments in dense stereo matching, confidence estimation is beneficial for a wide range of applications, e.g. autonomous driving, which needs a certain degree of confidence as mandatory prerequisite. Especially, the introduction of deep learning based methods resulted in an increasing popularity of this field in recent years, caused by a significantly improved accuracy. Despite this remarkable development, most of these methods rely on features learned from disparity maps only, not taking into account the corresponding 3-dimensional cost volumes. However, it was already demonstrated that with conventional methods based on hand-crafted features this additional information can be used to further increase the accuracy. In order to combine the advantages of deep learning and cost volume based features, in this paper, we propose a novel Convolutional Neural Network (CNN) architecture to directly learn features for confidence estimation from volumetric 3D data. An extensive evaluation on three datasets using three common dense stereo matching techniques demonstrates the generality and state-of-the-art accuracy of the proposed method.