Real Time Dense Depth Estimation by Fusing Stereo with Sparse Depth Measurements

Abstract

We present an approach to depth estimation that fuses information from a stereo pair with sparse range measurements derived from a LIDAR sensor or a range camera. The goal of this work is to exploit the complementary strengths of the two sensor modalities, the accurate but sparse range measurements and the ambiguous but dense stereo information. These two sources are effectively and efficiently fused by combining ideas from anisotropic diffusion and semi-global matching.We evaluate our approach on the KITTI 2015 and Middlebury 2014 datasets, using randomly sampled ground truth range measurements as our sparse depth input. We achieve significant performance improvements with a small fraction of range measurements on both datasets. We also provide qualitative results from our platform using the PMDTec Monstar sensor. Our entire pipeline runs on an NVIDIA TX-2 platform at 5Hz on 1280×1024 stereo images with 128 disparity levels.

3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

“…Thanks to the complementary characteristic across different sensors, several works [1] [2] have studied how to fuse multiple modalities in order to provide more accurate and denser depth estimation. In this paper, we consider the fusion of passive stereo camera and active 3D LiDAR sensor, which is a practical and popular choice.…”

LiDAR-guided stereo matching with a spatial consistency constraint

“…The Gaussian enhancement function was used to optimize the cost volume on each LiDAR projection point, which improved the accuracy of dense matching (Poggi et al, 2019). Furthermore, expanding the LiDAR projection points yielded much improved dense matching results by predicting the disparities of the neighboring pixels around the LiDAR projection points (Shivakumar et al, 2019); however, its matching accuracy relies on the accuracy of the predicted disparities.…”

“…Expanding the number of LiDAR projection points by predicting the disparities of the neighboring pixels around the LiDAR projection points can overcome the above problems (Shivakumar et al, 2019). But this approach also brings a new challenge that the dense matching results rely on the accuracy of the predicted disparities.…”

A survey on visual servoing for wheeled mobile robots

“…For example, measurements from the IMU can be aligned with image feedback to recover metric information like Qin et al (2018). The vision system can also be combined with LiDAR sensors, and sparse laser points help to achieve dense perception (Shivakumar et al 2019).…”