Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo

Maddern, Will; Newman, Paul

doi:10.1109/iros.2016.7759342

Cited by 84 publications

(68 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method successfully recovers accurate disparities on tiny and moving objects while the other methods are misled by drifted and noisy Lidar points. sion studies [2,20,26] strongly depend on the availability of large-scale ground truth depth maps, and thus their performance is fundamentally limited by their generalization ability to real-world applications.…”

Section: Input Lidarmentioning

confidence: 99%

“…Our plane fitting loss then can be defined as (e)S2D [19] (f) SINet [30] (g) Probabilistic fusion [20] (h) CNN fusion [26] Figure 5. Qualitative results of the methods from Tab.…”

Section: Plane Fitting Lossmentioning

confidence: 99%

“…To obtain sparse disparities inputs, we projected raw Lidar points onto left and right images using provided extrinsic and intrinsic parameters and converted the raw Lidar depths to disparities. Maddern et al [20] also traced 141 frames from KITTI raw dataset that have corresponding frames in the KITTI 2015 dataset and reported their results on this subset. For consistency, we used the same subset to evaluate our performance and utilize the 6 frames from KITTI VO dataset as our validation set (we excluded 1 frame that overlaps the KITTI 141 subset from our validation).…”

Section: Kitti Datasetmentioning

confidence: 99%

See 2 more Smart Citations

Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

Cheng

Zhong

Dai

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

In this paper, we present LidarStereoNet, the first unsupervised Lidar-stereo fusion network, which can be trained in an end-to-end manner without the need of ground truth depth maps. By introducing a novel "Feedback Loop" to connect the network input with output, LidarStereoNet could tackle both noisy Lidar points and misalignment between sensors that have been ignored in existing Lidarstereo fusion studies. Besides, we propose to incorporate a piecewise planar model into network learning to further constrain depths to conform to the underlying 3D geometry. Extensive quantitative and qualitative evaluations on both real and synthetic datasets demonstrate the superiority of our method, which outperforms state-of-the-art stereo matching, depth completion and Lidar-Stereo fusion approaches significantly.

show abstract

Section: Input Lidarmentioning

confidence: 99%

“…Our plane fitting loss then can be defined as (e)S2D [19] (f) SINet [30] (g) Probabilistic fusion [20] (h) CNN fusion [26] Figure 5. Qualitative results of the methods from Tab.…”

Section: Plane Fitting Lossmentioning

confidence: 99%

Section: Kitti Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

Cheng

Zhong

Dai

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…As an alternative to deep neural networks, graphical models have been successfully applied to point clouds for various tasks. Previous works by Maddern & Newman [26] and Schoenberg et al [27] use graphical models to fuse lidar scans and stereo camera depth maps to produce accurate dense depth maps suitable for use on autonomous vehicles. Wang et al [28] propose a semantic segmentation method for image-aligned 3D point clouds by retrieving referenced labeled images of similar appearances and then propagating their labels to the 3D points using a graphical model.…”

Section: B Graphical Models and 2d-3d Fusionmentioning

confidence: 99%

LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images

Wang

Chao

Wang

et al. 2019

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Object segmentation in 3D point clouds is a critical task for robots capable of 3D perception. Despite the impressive performance of deep learning-based approaches on object segmentation in 2D images, deep learning has not been applied nearly as successfully for 3D point cloud segmentation. Deep networks generally require large amounts of labeled training data, which are readily available for 2D images but are difficult to produce for 3D point clouds. In this paper, we present Label Diffusion Lidar Segmentation (LDLS), a novel approach for 3D point cloud segmentation which leverages 2D segmentation of an RGB image from an aligned camera to avoid the need for training on annotated 3D data. We obtain 2D segmentation predictions by applying Mask-RCNN to the RGB image, and then link this image to a 3D lidar point cloud by building a graph of connections among 3D points and 2D pixels. This graph then directs a semi-supervised label diffusion process, where the 2D pixels act as source nodes that diffuse object label information through the 3D point cloud, resulting in a complete 3D point cloud segmentation. We conduct empirical studies on the KITTI benchmark data set and on a mobile robot, demonstrating wide applicability and superior performance of LDLS compared to the previous state-of-the-art in 3D point cloud segmentation, without any need for either 3D training data or fine-tuning of the 2D image segmentation model.

show abstract

“…In recent years, different kinds of depth fusion methods have emerged in different sub-tasks, such as stereo-ToF fusion ( [6,7,8]), stereo-stereo fusion ( [9]), Lidar-stereo fusion ( [10,11]) and general depth fusion ( [12]). Additionally, deep-learning based methods perform much better than the rest.…”

Section: Introductionmentioning

confidence: 99%

UDFNET: Unsupervised Disparity Fusion with Adversarial Networks

Fisher

2019

2019 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Existing disparity fusion methods based on deep learning achieve state-of-the-art performance, but they require ground truth disparity data to train. As far as I know, this is the first time an unsupervised disparity fusion not using ground truth disparity data has been proposed. In this paper, a mathematical model for disparity fusion is proposed to guide an adversarial network to train effectively without ground truth disparity data. The initial disparity maps are inputted from the left view along with auxiliary information (gradient, left & right intensity image) into the refiner and the refiner is trained to output the refined disparity map registered on the left view. The refined left disparity map and left intensity image are used to reconstruct a fake right intensity image. Finally, the fake and real right intensity images (from the right stereo vision camera) are fed into the discriminator. In the model, the refiner is trained to output a refined disparity value close to the weighted sum of the disparity inputs for global initialisation. Then, three refinement principles are adopted to refine the results further. (1) The reconstructed intensity error between the fake and real right intensity image is minimised.(2) The similarities between the fake and real right image in different receptive fields are maximised.(3) The refined disparity map is smoothed based on the corresponding intensity image. The adversarial networks' architectures are effective for the fusion task. The fusion time using the proposed network is small. The network can achieve 90 fps using Nvidia Geforce GTX 1080Ti on the Kitti2015 dataset when the input resolution is 1242 * 375 (Width * Height) without downsampling and cropping. The accuracy of this work is equal to (or better than) the state-of-the-art supervised methods.

show abstract

Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo

Cited by 84 publications

References 21 publications

Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images

UDFNET: Unsupervised Disparity Fusion with Adversarial Networks

Contact Info

Product

Resources

About