Hierarchical Deep Stereo Matching on High-Resolution Images

Abstract

Figure 1: Illustration of on-demand depth sensing with a coarse-to-fine hierarchy on the proposed dataset. Our method (HSM) captures the coarse layout of the scene in 91 milliseconds, finds the far-away car (shown in the red box) in 175 ms, and recovers the details of the car given extra 255 ms. AbstractWe explore the problem of real-time stereo matching on high-res imagery. Many state-of-the-art (SOTA) methods struggle to process high-res imagery because of memory constraints or speed limitations. To address this issue, we propose an end-to-end framework that searches for correspondences incrementally over a coarse-to-fine hierarchy. Because high-res stereo datasets are relatively rare, we introduce a dataset with high-res stereo pairs for both training and evaluation. Our approach achieved SOTA performance on Middlebury-v3 and KITTI-15 while running significantly faster than its competitors. The hierarchical design also naturally allows for anytime on-demand reports of disparity by capping intermediate coarse results, allowing us to accurately predict disparity for near-range structures with low latency (30ms). We demonstrate that the performance-vs-speed tradeoff afforded by on-demand hierarchies may address sensing needs for time-critical applications such as autonomous driving.

A Lightweight Depth Estimation Network for Wide-Baseline Light Fields

“…It is clear that our estimated disparity maps are all closer to the ground truth and have the fewest number of bad pixels. In contrast, the estimated disparity maps from LF OCC are noisy, those from RPRF look over-smoothed and have quantification errors, both EPINET and EPINET T fail to predict disparities, HSM [21] is disturbed by the ambiguous background, EPI-Shift [26] and LBDE-E [25] both seem not able to handle the foreground well.…”

“…To verify the effectiveness of our network LLF-Net on widebaseline light field datasets, we conducted experiments on the synthetic WLF dataset and real-world Google [43] and ULB [44] datasets. The proposed LLF-Net is compared with recent state-of-the-art depth estimation methods, comprising of traditional light field depth estimation methods LF OCC [3] and RPRF [13], ConvNet-based methods HSM [21], EPINET [19] and LBDE-E [25]. With respect to ConvNet-based methods, HSM [21] is designed for stereo-based depth estimation, EPINET [19] achieves top performance among published methods in narrow-baseline light field datasets and LBDE-E [25] achieves considerable performance on both narrowand wide-baseline light field datasets.…”

“…The proposed LLF-Net is compared with recent state-of-the-art depth estimation methods, comprising of traditional light field depth estimation methods LF OCC [3] and RPRF [13], ConvNet-based methods HSM [21], EPINET [19] and LBDE-E [25]. With respect to ConvNet-based methods, HSM [21] is designed for stereo-based depth estimation, EPINET [19] achieves top performance among published methods in narrow-baseline light field datasets and LBDE-E [25] achieves considerable performance on both narrowand wide-baseline light field datasets. Note that EPINET [19] is originally trained on narrow-baseline datasets and fails to infer depth in wide-baseline datasets, therefore we re-trained it using their public source code, and denote it as EPINET T. VII shows the quantitative comparisons on the four exemplar scenes from Hand-designed test set of the WLF dataset.…”

“…The baseline between the recorded wide-baseline light-field images is large and the spatial resolution of images is usually high. Compared with the narrow-baseline scenario, the wide-baseline is more capable of improving depth accuracy due to its large baseline [21]. Up to now, considerable efforts have been also made by traditional methods [3,6,10,13,[22][23][24] to solve the problem of depth estimation in the wide-baseline scenario.…”

MPA‐Net: multi‐path attention stereo matching network

“…In EdgeStereo [21], the general decoder structure was replaced by using the new designed residual pyramid network with different scales. The HSM [46] computed the encoder features in a coarse‐to‐fine with 3D residual convolution and volumetric pyramid pooling, and gradually increased the output resolution. It could predict accurate disparity map with low latency.…”