Domain-Invariant Stereo Matching Networks

Zhang, Feihu; Qi, Xiaojuan; Yang, Ruigang; Prisacariu, Victor Adrian; Wah, Benjamin W.; Torr, Philip H. S.

doi:10.1007/978-3-030-58536-5_25

Cited by 122 publications

(81 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To overcome this problem, Zhang et al [52] proposed DSMNet, which employs Domain Normalization and non-local graph-based filtering layers to enforce the learning of structural features that are domain-invariant. Similarly, Shen et al [36] introduced CFNet, an efficient network architecture with multi-scale cost volume fusion and refinement, to enforce the learning of robust structural representation for stereo matching.…”

Section: Learning-based Stereo Matching Networkmentioning

confidence: 99%

“…and L smooth L1 is the smooth-L1 loss function commonly employed for optimizing stereo matching networks [5,15,51,52].…”

Section: Approximating Fisher Informationmentioning

confidence: 99%

“…In Tab. 4, we compare the synthetic-to-realistic domain generalization performance of our method with the state-ofthe-art stereo matching networks [5,14,15,36,48,51,52] on the four realistic datasets. All networks are trained on the synthetic Scene Flow training set only.…”

Section: Synthetic-to-realistic Domain Generalization Evaluationmentioning

confidence: 99%

“…The proposed ITSA is computationally efficient, and as we show by extensive experiments, it can promote the learning of shortcut-invariant features. Unlike the existing domaininvariant stereo matching networks [36,52], the proposed ITSA does not involve significant network alteration and is model-agnostic. Therefore, as shown in the experiments section, it can be easily integrated with different stereo matching networks.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Chuah¹,

Tennakoon²,

Hoseinnezhad³

et al. 2022

Preprint

View full text Add to dashboard Cite

State-of-the-art stereo matching networks trained only on synthetic data often fail to generalize to more challenging real data domains. In this paper, we attempt to unfold an important factor that hinders the networks from generalizing across domains: through the lens of shortcut learning. We demonstrate that the learning of feature representations in stereo matching networks is heavily influenced by synthetic data artefacts (shortcut attributes). To mitigate this issue, we propose an Information-Theoretic Shortcut Avoidance (ITSA) approach to automatically restrict shortcutrelated information from being encoded into the feature representations. As a result, our proposed method learns robust and shortcut-invariant features by minimizing the sensitivity of latent features to input variations. To avoid the prohibitive computational cost of direct input sensitivity optimization, we propose an effective yet feasible algorithm to achieve robustness. We show that using this method, stateof-the-art stereo matching networks that are trained purely on synthetic data can effectively generalize to challenging and previously unseen real data scenarios. Importantly, the proposed method enhances the robustness of the synthetic trained networks to the point that they outperform their finetuned counterparts (on real data) for challenging out-ofdomain stereo datasets.

show abstract

Section: Learning-based Stereo Matching Networkmentioning

confidence: 99%

“…and L smooth L1 is the smooth-L1 loss function commonly employed for optimizing stereo matching networks [5,15,51,52].…”

Section: Approximating Fisher Informationmentioning

confidence: 99%

Section: Synthetic-to-realistic Domain Generalization Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Chuah¹,

Tennakoon²,

Hoseinnezhad³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This allows the network to learn the entire model end to end while leveraging our geometric knowledge of the stereo problem. What's more, the current learning-based stereo matching models (Zhang, Prisacariu, Yang & Torr 2019, Zhang, Qi, Yang, Prisacariu, Wah & Torr 2020 can effectively perform migration training by being pre-trained in the synthetic dataset (Mayer, Ilg, Häusser, Fischer, Cremers, Dosovitskiy & Brox 2016) and finetuning in the application dataset with limited ground truth depth data. Inspired by these ideas, we design a "Ground Truth" labels module based on a stereo matching model and pose method to generate depth and pose data offline as weakly supervised labels.…”

Section: A C C E P T E D U N E D I T E D M Smentioning

confidence: 99%

Monocular weakly supervised depth and pose estimation method based on multi-information fusion

Zhang

Qiao

Lin

2021

KJS publishes peer-review articles in Mathematics, Computer Sci

View full text Add to dashboard Cite

The depth and pose information are the basic issues in the ﬁeld of robotics, autonomous driving, and virtual reality, and are also the focus and difﬁcult issues of computer vision research. The supervised monocular depth and pose estimation learning are not feasible in environments where labeled data is not abundant. Self-supervised monocular video methods can learn effectively only by applying photometric constraints without expensive ground true depth label constraints, which results in an inefﬁcient training process and suboptimal estimation accuracy. To solve these problems, a monocular weakly supervised depth and pose estimation method based on multi-information fusion is proposed in this paper. First, we design a high-precision stereo matching method to generate a depth and pose data as the "Ground Truth" labels to solve the problem that the ground truth labels are difﬁcult to obtain. Then, we construct a multi-information fusion network model based on the "Ground truth" labels, video sequence, and IMU information to improve the estimation accuracy. Finally, we design the loss function of supervised cues based on "Ground Truth" labels cues and self-supervised cues to optimize our model. In the testing phase, the network model can separately output high-precision depth and pose data from a monocular video sequence. The resulting model outperforms mainstream monocular depth and poses estimation methods as well as the partial stereo matching method in the challenging KITTI dataset by only using a small number of real training data(200 pairs).

show abstract

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Tosi

Poggi

Benincasa

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Confidence measures for stereo gained popularity in recent years due to their improved capability to detect outliers and the increasing number of applications exploiting these cues. In this field, convolutional neural networks achieved top-performance compared to other known techniques in the literature by processing local information to tell disparity assignments from outliers. Despite this outstanding achievements, all approaches rely on clues extracted with small receptive fields thus ignoring most of the overall image content. Therefore, in this paper, we propose to exploit nearby and farther clues available from image and disparity domains to obtain a more accurate confidence estimation. While local information is very effective for detecting high frequency patterns, it lacks insights from farther regions in the scene. On the other hand, enlarging the receptive field allows to include clues from farther regions but produces smoother uncertainty estimation, not particularly accurate when dealing with high frequency patterns. For these reasons, we propose in this paper a multi-stage cascaded network to combine the best of the two worlds. Extensive experiments on three datasets using three popular stereo algorithms prove that the proposed framework outperforms state-of-the-art confidence estimation techniques.

show abstract

Domain-Invariant Stereo Matching Networks

Cited by 122 publications

References 56 publications

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Monocular weakly supervised depth and pose estimation method based on multi-information fusion

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Contact Info

Product

Resources

About