PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimation

Abstract

In the last few years, convolutional neural networks (CNNs) have demonstrated increasing success at learning many computer vision tasks including dense estimation problems such as optical flow and stereo matching. However, the joint prediction of these tasks, called scene flow, has traditionally been tackled using slow classical methods based on primitive assumptions which fail to generalize. The work presented in this paper overcomes these drawbacks efficiently (in terms of speed and accuracy) by proposing PWOC-3D, a compact CNN architecture to predict scene flow from stereo image sequences in an end-to-end supervised setting. Further, large motion and occlusions are well-known problems in scene flow estimation. PWOC-3D employs specialized design decisions to explicitly model these challenges. In this regard, we propose a novel selfsupervised strategy to predict occlusions from images (learned without any labeled occlusion data). Leveraging several such constructs, our network achieves competitive results on the KITTI benchmark and the challenging FlyingThings3D dataset. Especially on KITTI, PWOC-3D achieves the second place among end-to-end deep learning methods with 48 times fewer parameters than the top-performing method.

Learning End-to-End Scene Flow by Distilling Single Tasks Knowledge

“…Specifically, the full search space of such operation is 3D, being it over pixel coordinates plus displacement between correlation curves. We refer the reader to the supplementary material for some examples supporting this rationale, while ablation studies reported among our experiments prove the effectiveness of such a new layer, allowing DWARF to outperform similar architectures (Saxena et al 2019) on the KITTI online benchmark.…”

“…Recently, Saxena et al (2019) proposed a fast and lightweight model taking into account occlusions. Although similar in design, we will show that DWARF outperforms it by a good margin.…”

SS-SF: Piecewise 3D Scene Flow Estimation With Semantic Segmentation

“…In recent years, the CNN-based scene flow methods have shown the good performance on both computational accuracy and efficiency [57]- [59]. However, most of these CNN-based usually approaches require supervised training process and may have difficulty to be directly applied to real world data where ground truth is not easily accessible.…”

SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation

“…A rigid plane model performs poorly when applied to deformable objects, and ego-motion estimation for highly dynamic scenes is hard. (Menze and Geiger, 2015) CSF (Lv et al, 2016) Deep Learning Fast Poor generalization PWOC-3D (Saxena et al, 2019) DRISF (Ma et al, 2019) Sparse-to-Dense Comparatively fast, good generalization Sensitive to distribution of matches SFF Schuster et al (2018c) SFF++ (ours)…”

“…Methods which are guided by semantic segmentation from deep neural networks will generalize badly to other domains, unless they are fine-tuned for the new task. Same is assumed for upcoming purely learning based approaches (Ma et al, 2019;Saxena et al, 2019) which are potentially even faster than our approach. SFF++ focuses especially on robustness across domains and applications.…”