DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints

Han, Liming; Lin, Yaping; Du, Guoguang; Lian, Shiguo

doi:10.48550/arxiv.1906.11435

Cited by 7 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For supervised learning VO methods, these approaches infer the camera pose by learning directly from real image data, such as Flowdometry [43], cast the VO problem as a regression problem by using FlowNet [44] to extract optical flow features and a fully connected layer to predict camera translation and rotation, and DVO [13] and ESP-VO [45] incorporate recurrent neural networks (RNNs), to implicitly model the sequential motion dynamics of the image sequences. Han, L. et al [13] presented a self-supervised deep learning network for monocular VIO; Shamwell et al [46] presented an unsupervised deep neural network approach to the fusion of RGB-D imagery with inertial measurements for absolute trajectory estimation. Inspired by this work, we incorporated raw IMU data into a visual, odometry-based deep keypoint with a fusion model to regularize the camera pose and alignment depth map.…”

Section: Deep Visual-inertial Odometry Learning Methodsmentioning

confidence: 99%

“…Both extensions improve the baseline and the attention module performs well. When coupled with the self-supervised depth estimation, the DeepVIO performance training-to have a consistent pose estimation-outperforms all of the state-of-the-art, compared to classical SLAM libviso2 and learning-based techniques, Sc-SfMLearner and SfMLearner [5,13,62]. Figures 6 and 7 show the trajectory in the XY-plane.…”

Section: Pose Estimationmentioning

confidence: 99%

“…To integrate visual odometry (VO) or the SLAM system into depth estimation, the authors of [10,12,13] presented a neural network to correct classical VO estimators in a selfsupervised manner and enhance geometric constraints. Self-supervised depth estimation, using the pose and depth between two adjacent frames, establishes a depth reprojection error and image reconstruction error [14][15][16][17].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

et al. 2022

View full text Add to dashboard Cite

This paper presents a new deep visual-inertial odometry and depth estimation framework for improving the accuracy of depth estimation and ego-motion from image sequences and inertial measurement unit (IMU) raw data. The proposed framework predicts ego-motion and depth with absolute scale in a self-supervised manner. We first capture dense features and solve the pose by deep visual odometry (DVO), and then combine the pose estimation pipeline with deep inertial odometry (DIO) by the extended Kalman filter (EKF) method to produce the sparse depth and pose with absolute scale. We then join deep visual-inertial odometry (DeepVIO) with depth estimation by using sparse depth and the pose from DeepVIO pipeline to align the scale of the depth prediction with the triangulated point cloud and reduce image reconstruction error. Specifically, we use the strengths of learning-based visual-inertial odometry (VIO) and depth estimation to build an end-to-end self-supervised learning architecture. We evaluated the new framework on the KITTI datasets and compared it to the previous techniques. We show that our approach improves results for ego-motion estimation and achieves comparable results for depth estimation, especially in the detail area.

show abstract

Section: Deep Visual-inertial Odometry Learning Methodsmentioning

confidence: 99%

Section: Pose Estimationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

et al. 2022

View full text Add to dashboard Cite

show abstract

“…VIOLearner [49] presents an online error correction module for deep visual-inertial odometry that estimates the trajectory by fusing RGB-D images with inertial data. DeepVIO [18] recently proposed a fusion network to fuse visual and inertial features. This network is trained with a dedicated loss.…”

Section: Learning-based Pose Estimationmentioning

confidence: 99%

Learning Selective Sensor Fusion for States Estimation

Chen¹,

Rosa²,

Lu³

et al. 2019

Preprint

View full text Add to dashboard Cite

Autonomous vehicles and mobile robotic systems are typically equipped with multiple sensors to provide redundancy. By integrating the observations from different sensors, these mobile agents are able to perceive the environment and estimate system states, e.g. locations and orientations. Although deep learning approaches for multimodal odometry estimation and localization have gained traction, they rarely focus on the issue of robust sensor fusion -a necessary consideration to deal with noisy or incomplete sensor observations in the real world. Moreover, current deep odometry models also suffer from a lack of interpretability. To this extent, we propose SelectFusion, an end-to-end selective sensor fusion module which can be applied to useful pairs of sensor modalities such as monocular images and inertial measurements, depth images and LIDAR point clouds. During prediction, the network is able to assess the reliability of the latent features from different sensor modalities and estimate both trajectory at scale and global pose. In particular, we propose two fusion modules based on different attention strategies: deterministic soft fusion and stochastic hard fusion, and we offer a comprehensive study of the new strategies compared to trivial direct fusion. We evaluate all fusion strategies in both ideal conditions and on progressively degraded datasets that present occlusions, noisy and missing data and time misalignment between sensors, and we investigate the effectiveness of the different fusion strategies in attending the most reliable features, which in itself, provides insights into the operation of the various models.

show abstract

“…In case of a localization problem as an example, learning from complementary sensor types such as cameras and IMUs can bring improved robustness and redundancy when there exists sensor corruptions and failures. Existing methods for learning based VIO rely on different strategies to fuse visual and inertial data, which include so-called FC-fusion networks (Clark et al, 2017b;Han et al, 2019) and soft/hard feature selection networks (Chen et al, 2019;Almalioglu et al, 2019). Broadly speaking, these strategies typically learn additive interac-tions for integrating multiple streams of information, i.e.…”

Section: Introductionmentioning

confidence: 99%

Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry

Shinde,

Lee,

Humt

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper presents an end-to-end multi-modal learning approach for monocular Visual-Inertial Odometry (VIO), which is specifically designed to exploit sensor complementarity in the light of sensor degradation scenarios. The proposed network makes use of a multi-head self-attention mechanism that learns multiplicative interactions between multiple streams of information. Another design feature of our approach is the incorporation of the model uncertainty using scalable Laplace Approximation. We evaluate the performance of the proposed approach by comparing it against the end-to-end state-of-the-art methods on the KITTI dataset and show that it achieves superior performance. Importantly, our work thereby provides an empirical evidence that learning multiplicative interactions can result in a powerful inductive bias for increased robustness to sensor failures.

show abstract

DeepVIO: Self-supervised Deep Learning of Monocular Visual Inertial Odometry using 3D Geometric Constraints

Cited by 7 publications

References 30 publications

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

Learning Selective Sensor Fusion for States Estimation

Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry

Contact Info

Product

Resources

About