DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation From Stereo Imagery

Abstract

Recent work has shown that convolutional neural networks (CNNs) can be applied successfully in disparity estimation, but these methods still suffer from errors in regions of low-texture, occlusions and reflections. Concurrently, deep learning for semantic segmentation has shown great progress in recent years. In this paper, we design a CNN architecture that combines these two tasks to improve the quality and accuracy of disparity estimation with the help of semantic segmentation. Specifically, we propose a network structure in which these two tasks are highly coupled. One key novelty of this approach is the two-stage refinement process. Initial disparity estimates are refined with an embedding learned from the semantic segmentation branch of the network. The proposed model is trained using an unsupervised approach, in which images from one half of the stereo pair are warped and compared against images from the other camera. Another key advantage of the proposed approach is that a single network is capable of outputting disparity estimates and semantic labels. These outputs are of great use in autonomous vehicle operation; with real-time constraints being key, such performance improvements increase the viability of driving applications. Experiments on KITTI and Cityscapes datasets show that our model can achieve state-ofthe-art results and that leveraging embedding learned from semantic segmentation improves the performance of disparity estimation.

Multi-Dimensional Residual Dense Attention Network for Stereo Matching

“…Displets [18] uses object knowledge that obtained by modeling 3D vehicles to resolve the stereo ambiguities for disparity estimation. The SegStereo [19] and DispSegNet [20] incorporate prior semantic information into stereo matching. The EdgeStereo [21] borrows the network architecture of edge detector HED [22] and achieves the mutual promotion between the edge detection task and stereo matching task.…”

Real-Time Semantic Stereo Matching

“…To the best of our knowledge, only four methods successfully achieved this result: SegStereo [37] and DispSegNet [38] leverage on segmentation as auxiliary task, AMNet [39] on Background foreground segmentation and EdgeStereo [40] that used edges cues. To better understand the proposed con- Figure 2.11: Architecture overview of SegStereo.…”

“…As this stage, the network would be able to provide in a parallel fashion disparity and segmentation outputs, anyway, a further and deeper exploitation of the segmentation cues is possible and it can be achieved with a specific refinement module. Influenced both by the AnyNet residual structure and by previous successful works [37,38] we decided to merge disparity cost volumes and semantic embeddings into hybrid volumes. Differently from other works though, we want to keep also this part of the architecture fully residual and pyramidal.…”

“…This means they own deeper and more meaningful representations regarding one of the two tasks. Recent work [37,38] proved that semantic features are a better descriptor for some image details and therefore can help the disparity regression task. Edges and corners, for example, might be easily inferred by a segmentation network especially when they divide two different classes.…”

Self-supervised Object Motion and Depth Estimation from Video

“…The image is synthesized from the network outputs, following the traditional Structure-frommotion procedure. Extra constraint and additional information have been introduced to improve the performance, like the temporal depth consistency [22], the stereo matching [23] and the semantic information [34]. Godard et al [11] achieved a significant improvement by compensating for image occlusion.…”