Self-supervised Visual-LiDAR Odometry with Flip Consistency

Abstract

Most learning-based methods estimate ego-motion by utilizing visual sensors, which suffer from dramatic lighting variations and textureless scenarios. In this paper, we incorporate sparse but accurate depth measurements obtained from lidars to overcome the limitation of visual methods. To this end, we design a self-supervised visual-lidar odometry (Self-VLO) framework. It takes both monocular images and sparse depth maps projected from 3D lidar points as input, and produces pose and depth estimations in an end-to-end learning manner, without using any ground truth labels. To effectively fuse two modalities, we design a two-pathway encoder to extract features from visual and depth images and fuse the encoded features with those in decoders at multiple scales by our fusion module. We also adopt a siamese architecture and design an adaptively weighted flip consistency loss to facilitate the selfsupervised learning of our VLO. Experiments on the KITTI odometry benchmark show that the proposed approach outperforms all self-supervised visual or lidar odometries. It also performs better than fully supervised VOs, demonstrating the power of fusion.

Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth

“…Instead of direct regression model for ego-motion, [26] proposes to estimate an optical flow between two images and solves ego-motion as an optimal fundamental matrix. Besides these works which still perform ego-motion estimation purely in the RGB domain, some other works attempt to introduce data from other sensors, e.g., LiDAR [3] and IMU [14,16], as additional inputs to improve the ego-motion accuracy in the spirit of sensor fusion. Following the idea of performing motion estimation in a mixed domain, [8] further proposes a two-stream network that leverages the original RGB images and the internally inferred depth maps as inputs of egomotion network.…”

“…One key component in them is to get an accurate ego-motion between two consecutive frames, which is often carried as camera pose estimation using RGB images. On top of the great success of classical methods based on 3D geometry and camera models, learning-based pose estimation methods [1,2] have recently got increasing research interests for their good fits to training data and feasibility in typical severe situations, e.g., poor lighting [3]. Most of these works treat the pose estimation problem as a regression from input color images, and design pose estimators based on the convolutional neural network (CNN) [1,2,[4][5][6][7][8] or the recurrent neural network (RNN) [9][10][11][12].…”

“…One common challenge of those learned pose estimators is the capability to generalize to different situations, e.g., longterm scene changes including lighting changes [13]. Several works [3,[14][15][16] tackle this problem by adopting different modalities from additional sensors, such as LiDARs [3] and IMUs [14][15][16], along with RGB images. Alternatively, selfsupervised training strategies for the pose estimator have recently been studied [17][18][19] to utilize much more training data, which could consequently lead to a more generalized model.…”

“…Furthermore, recent works [8,10,12,20] reported the fact that feeding a 1 Dept. of Systems and Control Engineering, Tokyo Institute of Technology, {zjiang, htaira, mxo}@ok.sc.e.titech.ac.jp 2 R&D Group, Olympus, naoyuki.miyashita@olympus.com 3 Our code will be available at https://github.com/Beniko95J/MLF-VO. pose estimator not only with original images but also with depth images predicted from RGB images can improve the performance, as well as fusing different modalities from real sensors.…”

What's in My LiDAR Odometry Toolbox?

“…Others combine features from different sources or sensors. [12] uses a Siamese pyramid network which combines features from two pairs of sparse depth and RGB images and regresses the relative motion, and depth.…”