Deep Depth Estimation from Visual-Inertial SLAM

Sartipi, Kourosh; Do, Tien Van; Ke, Tong; Vuong, Khiem; Roumeliotis, Stergios I.

doi:10.1109/iros45743.2020.9341448

Cited by 29 publications

(24 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depth completion: One approach towards dense depth estimation from multiple views is to: (i) Create a sparse point cloud (by tracking distinct 2D points across images and triangulating their 3D positions) and then (ii) Employ a depth-completion neural network that takes the sparse depth image along with the RGB image as inputs and exploits the scene's context to create a dense-depth estimate (e.g., [6], [7], [8], [9], [10]). Although these approaches have relatively low processing requirements, they are typically sensitive to the inaccuracies and sparsity level of their depth input; thus, they often fail to produce accurate depth estimates in textureless regions that lack sparse depth information.…”

Section: Related Work Multi-view Depth-estimation Methods Can Be Clas...mentioning

confidence: 99%

“…Specifically, approaches such as [4], [5] predict dense depth from a single image by taking advantage of images' contextual cues learned from large datasets; hence, they rely less on texture, as compared to classical methods. Moreover, to overcome the scale issue of singleview methods, depth-completion networks (e.g., [6], [7], [8], [9], [10]) leverage sparse point clouds from classical methods and complete the dense depth map using single-view cues. In order to further exploit multi-view information, depthestimation networks taking multiple images as input have also been considered.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Multi-view Depth Estimation with Predicted Uncertainty

Ke¹,

Do²,

Vuong³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address the problem of estimating dense depth from a sequence of images using deep neural networks. Specifically, we employ a dense-optical-flow network to compute correspondences and then triangulate the point cloud to obtain an initial depth map. Parts of the point cloud, however, may be less accurate than others due to lack of common observations or small baseline-to-depth ratio.To further increase the triangulation accuracy, we introduce a depth-refinement network (DRN) that optimizes the initial depth map based on the image's contextual cues. In particular, the DRN contains an iterative refinement module (IRM) that improves the depth accuracy over iterations by refining the deep features. Lastly, the DRN also predicts the uncertainty in the refined depths, which is desirable in applications such as measurement selection for scene reconstruction. We show experimentally that our algorithm outperforms state-of-the-art approaches in terms of depth accuracy, and verify that our predicted uncertainty is highly correlated to the actual depth error.

show abstract

Section: Related Work Multi-view Depth-estimation Methods Can Be Clas...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep Multi-view Depth Estimation with Predicted Uncertainty

Ke¹,

Do²,

Vuong³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Whereas, [8,7,32,33] learned uncertainty of estimates, [41] leveraged confidence maps, and [31,48,50] used surface normals for guidance. Like us, [27,36,52] proposed lightweight networks that can be deployed onto SLAM/VIO systems.…”

Section: Related Work and Contributionsmentioning

confidence: 99%

Unsupervised Depth Completion with Calibrated Backprojection Layers

Wong¹,

Soatto²

2021

Preprint

View full text Add to dashboard Cite

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KB-Net imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30% indoor and 8% outdoor when the same camera is used for training and testing. When the test camera is different, the improvement reaches 62%. Code available at: https://github.com/alexklwong/ calibrated-backprojection-network.

show abstract

“…To integrate visual odometry (VO) or the SLAM system into depth estimation, the authors of [10,12,13] presented a neural network to correct classical VO estimators in a selfsupervised manner and enhance geometric constraints. Self-supervised depth estimation, using the pose and depth between two adjacent frames, establishes a depth reprojection error and image reconstruction error [14][15][16][17]. In a monocular depth self-supervised estimation, the depth value estimated by the depth estimation network (DepthNet) and the pose between adjacent images have a decisive influence on the depth estimation result.…”

Section: Introductionmentioning

confidence: 99%

“…Motivated by these observations, we present a new deep visual-inertial odometry (DeepVIO) based ego-motion and depth prediction system that combines the strengths of learning-based VIO and geometrical depth estimation [16,19,20]. It uses DeepVIO geometrical constraints [21], where they are available, to achieve accurate odometry fusing with raw inertial measurement unit (IMU) data and sparse point clouds.…”

Section: Introductionmentioning

confidence: 99%

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

et al. 2022

View full text Add to dashboard Cite

This paper presents a new deep visual-inertial odometry and depth estimation framework for improving the accuracy of depth estimation and ego-motion from image sequences and inertial measurement unit (IMU) raw data. The proposed framework predicts ego-motion and depth with absolute scale in a self-supervised manner. We first capture dense features and solve the pose by deep visual odometry (DVO), and then combine the pose estimation pipeline with deep inertial odometry (DIO) by the extended Kalman filter (EKF) method to produce the sparse depth and pose with absolute scale. We then join deep visual-inertial odometry (DeepVIO) with depth estimation by using sparse depth and the pose from DeepVIO pipeline to align the scale of the depth prediction with the triangulated point cloud and reduce image reconstruction error. Specifically, we use the strengths of learning-based visual-inertial odometry (VIO) and depth estimation to build an end-to-end self-supervised learning architecture. We evaluated the new framework on the KITTI datasets and compared it to the previous techniques. We show that our approach improves results for ego-motion estimation and achieves comparable results for depth estimation, especially in the detail area.

show abstract

Deep Depth Estimation from Visual-Inertial SLAM

Cited by 29 publications

References 38 publications

Deep Multi-view Depth Estimation with Predicted Uncertainty

Deep Multi-view Depth Estimation with Predicted Uncertainty

Unsupervised Depth Completion with Calibrated Backprojection Layers

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

Contact Info

Product

Resources

About