Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth

Abstract

Self-supervised monocular scene flow estimation, aiming to understand both 3D structures and 3D motions from two temporally consecutive monocular images, has received increasing attention for its simple and economical sensor setup. However, the accuracy of current methods suffers from the bottleneck of less-efficient network architecture and lack of motion rigidity for regularization. In this paper, we propose a superior model named EMR-MSF by borrowing the advantages of network architecture design under the scope of supervised learning. We further impose explicit and robust geometric constraints with an elaborately constructed ego-motion aggregation module where a rigidity soft mask is proposed to filter out dynamic regions for stable ego-motion estimation using static regions. Moreover, we propose a motion consistency loss along with a mask regularization loss to fully exploit static regions. Several efficient training strategies are integrated including a gradient detachment technique and an enhanced view synthesis process for better performance. Our proposed method outperforms the previous self-supervised works by a large margin and catches up to the performance of supervised methods. On the KITTI scene flow benchmark, our approach improves the SF-all metric of the state-of-theart self-supervised monocular method by 44% and demonstrates superior performance across sub-tasks including depth and visual odometry, amongst other self-supervised single-task or multi-task methods.

CertainOdom: Uncertainty Weighted Multi-task Learning Model for LiDAR Odometry Estimation

“…Compared with VINS-Mono [23], the advantage of our proposed approach not only exists in fusion stage but also in front-end feature extraction which we mentioned in Section II.A. Besides, the improvement compared with EMA-VIO [1] which also deploys Transformer-based approach for fusion, is possibly that the multi-layer fusion module aggregates the LiDAR and inertial data at different scale [32] [31].…”

“…The number of multi-head is set to 4. Most of empirical hyperparameters are referenced from other related works, such as MLF-VO [32] and Transfuser [48].…”

“…If we feed into different regressors and then fuse them, it is defined as late fusion. Channel Exchange [31] and MLF-VO [32] have confirmed that multi-layer fusion is better than the previous three fusion positions. However, since the Transformer is a data-hungry model [33] compared to CNN-based models, implementing the Transformer with multi-layer fusion requires taking the over-fitting issue into consideration, which we discuss in the ablation study Section IV-C.…”

“…We design the ablation study by separating the modules and evaluating their impact on performance, as shown in Table IV. The multi-layer strategy has been validated in MLF-VO [32] and CE [31], and the multi-task regressor module is compared with the baseline 6D pose regressor in our previous work, CertainOdom [54]. Therefore, we keep these two modules as "open" status in all cases.…”

A review of visual SLAM with dynamic objects

“…Song et al (2021) used deep learning to obtain prior models of objects, combining object-level information with Visual SLAM and estimating object and camera poses through the extended Kalman filter. Jiang et al (2022) estimated relative pose by fusing depth maps and RGB maps, demonstrating the feasibility of this fusion approach using the KITTI data set. Zhou et al (2022a, 2022b) introduced local plane constraints to Visual SLAM, assuming local regions of flat ground, which finds applications in the field of autonomous driving.…”