Separable Convolutions for Optimizing 3D Stereo Networks

Rahim, Rafia; Shamsafar, Faranak; Zell, Andreas

doi:10.1109/icip42928.2021.9506330

Cited by 8 publications

(3 citation statements)

References 23 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, end‐to‐end fashion DNN solutions still suffer from unbalancing between low computation consumption and high‐quality depth estimation. For instance, Shamsafar et al (2022) and Rahim et al (2021) both identify that 3D convolution operations consume the most computational power and they provide alternative architectures to reduce the parameters and operations. However, the oversize models still prevent them from reaching real‐time requirements even on the powerful computing platform.…”

Section: Related Workmentioning

confidence: 99%

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Meng

Li²,

Zhong

et al. 2023

Journal of Field Robotics

View full text Add to dashboard Cite

Light detection and ranging (LiDAR) and stereo cameras are two generally used solutions for perceiving 3D information. The complementary properties of these two sensor modalities motivate a fusion to derive practicable depth sensing toward real‐world applications. Promoted by deep neural network (DNN) techniques, recent works achieve superior performance on accuracy. However, the complex architecture and the sheer number of DNN parameters often lead to poor generalization capacity and non‐real‐time computing. In this paper, we present FastFusion, a three‐stage stereo‐LiDAR deep fusion scheme, which integrates the LiDAR priors into each step of classical stereo‐matching taxonomy, gaining high‐precision dense depth sensing in a real‐time manner. We integrate stereo‐LiDAR information by taking advantage of a compact binary neural network and utilize the proposed cross‐based LiDAR trust aggregation to further fuse the sparse LiDAR measurements in the back‐end of stereo matching. To align the photometrical of the input image and the depth of the estimation, we introduce a refinement network to guarantee consistency. More importantly, we present a graphic processing unit‐based acceleration framework for providing a low‐latency implementation of FastFusion, gaining both accuracy improvement and real‐time responsiveness. In the experiments, we demonstrate the effectiveness and practicability of FastFusion, which obtains a significant speedup over state‐of‐the‐art baselines while achieving comparable accuracy on depth sensing. The video demo for real‐time depth estimation of FastFusion on the real‐world driving scenario is available at https://youtu.be/nP7cls2BA8s.

show abstract

Section: Related Workmentioning

confidence: 99%

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

Meng

Li²,

Zhong

et al. 2023

Journal of Field Robotics

View full text Add to dashboard Cite

show abstract

“…When faced with the ill-posed region challenges, SSPCV-Net [34] extracts details from semantic segmentation sub networks, the contentaware inter-scale cost aggregation method [35] adaptively aggregates and upsamples cost volume for reliable detail recovery, and MSMD-Net [36] constructs multi-scale and multi-dimension cost volume. Moreover, on the purpose of balancing real-time performance and accuracy, Gwc-Net [37] presents a group-wise correlation module that can not only provide similarity measurement, but also maintain better performance after reducing parameters, BGNet [38] proposes upsampling module based on the learned bilateral grid to get high quality cost volume form the low-resolution feature maps, and the method [39] realizes this goal by mean of the separable convolution. Targeting at earning a high level of accuracy, AcfNet [40] directly constraints the cost volume using true disparities peaked at unimodal distribution and the adaptive filtering cost volume.…”

Section: B Volumetric Stereo Matching Modelmentioning

confidence: 99%

HCVNet: Binocular Stereo Matching via Hybrid Cost Volume Computation Module With Attention

et al. 2022

View full text Add to dashboard Cite

Binocular stereo matching, a computer vision task typically using cost volume constructed from the left and right feature maps to estimate disparity and depth, is widely applied in 3D reconstruction, autonomous driving and robotics navigation. Though recent study brings an awareness of the convolution neural networks and the attention algorithms used in this field can make great progress, it is still difficult to satisfy the demand of high-precision applications due to many reasons. Study finds that the exist methods usually incline to ignore the intermediate feature map of other scales, pay less attention to the relationship between left and right feature maps and even just tend to use one type of cost volume to train the model. In this article, we mainly focus on solving the three problems mentioned above. Firstly, we present the Multi-scale Feature Extraction and Fusion Module(MFEFM) to get the informational feature maps via fusing all scale feature maps. And then we design the Effective Channel Attention Module(ECAM) applied to better capture and utilize the channel-wise independencies. Finally, we adopt the Hybrid Cost Volume Computation Module(HCVCM) to construct and aggregate cost volume. With these solutions, we build an end-to-end stereo matching network named HCVNet. Comparison with other state-of-the-art models, it can achieve 0.714px EPE on SceneFlow dataset, descending PSMNet(1.09px EPE) by 37.6%.INDEX TERMS binocular stereo matching, feature map, channel-wise independencies, channel attention, cost volume

show abstract

“…We achieve all of this without the need of costly 3D-convolutions or fullyconnected layers that are used by many popular state-of-the art methods such as GC-Net [4], PSMNet [5] or MC-CNNacrt [6]. That 3D-convolutions are a major bottleneck for stereo estimation networks has been shown by R. Rahim et al in their recent work [41].…”

Section: Introductionmentioning

confidence: 98%

FCDSN-DC: An Accurate and Lightweight Convolutional Neural Network for Stereo Estimation with Depth Completion

Hirner

Fraundorfer

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

We propose an accurate and lightweight convolutional neural network for stereo estimation with depth completion. We name this method fully-convolutional deformable similarity network with depth completion (FCDSN-DC). This method extends FC-DCNN by improving the feature extractor, adding a network structure for training highly accurate similarity functions and a network structure for filling inconsistent disparity estimates. The whole method consists of three parts. The first part consists of fully-convolutional densely connected layers that computes expressive features of rectified image pairs. The second part of our network learns highly accurate similarity functions between this learned features. It consists of densely-connected convolution layers with a deformable convolution block at the end to further improve the accuracy of the results. After this step an initial disparity map is created and the left-right consistency check is performed in order to remove inconsistent points. The last part of the network then uses this input together with the corresponding left RGB image in order to train a network that fills in the missing measurements. Consistent depth estimations are gathered around invalid points and are parsed together with the RGB points into a shallow CNN network structure in order to recover the missing values. We evaluate our method on challenging real world indoor and outdoor scenes, in particular Middlebury, KITTI and ETH3D were it produces competitive results. We furthermore show that this method generalizes well and is well suited for many applications without the need of further training.

show abstract

Separable Convolutions for Optimizing 3D Stereo Networks

Cited by 8 publications

References 23 publications

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

FastFusion: Deep stereo‐LiDAR fusion for real‐time high‐precision dense depth sensing

HCVNet: Binocular Stereo Matching via Hybrid Cost Volume Computation Module With Attention

FCDSN-DC: An Accurate and Lightweight Convolutional Neural Network for Stereo Estimation with Depth Completion

Contact Info

Product

Resources

About