Unsupervised Learning of Depth, Camera Pose and Optical Flow from Monocular Video

Abstract

We propose DFPNet -an unsupervised, joint learning system for monocular Depth, Optical Flow and egomotion (Camera Pose) estimation from monocular image sequences. Due to the nature of 3D scene geometry these three components are coupled. We leverage this fact to jointly train all the three components in an end-to-end manner. A single composite loss function -which involves image reconstruction-based loss for depth & optical flow, bidirectional consistency checks and smoothness loss components -is used to train the network. Using hyperparameter tuning, we are able to reduce the model size to less than 5% (∼8.4M parameters) of state-of-the-art DFP models. Evaluation on KITTI and Cityscapes driving datasets reveals that our model achieves results comparable to stateof-the-art in all of the three tasks, even with the significantly smaller model size.

GloFP-MSF: Monocular Scene Flow Estimation with Global Feature Perception

“…Estimating scene flow from monocular images sequence has been gaining increased attention. Manda et al [23] proposed DFPNet, which was an unsupervised joint learning system for monocular depth, optical flow and camera pose estimation from monocular image sequences. DRAFT [24] was a method that combined synthetic data with geometric self-supervision to jointly learn depth, optical flow and camera pose.…”

“…By utilizing advanced algorithms and techniques, scene flow estimation allows us to reconstruct and map the motion of objects across a sequence of images or frames, which has broad applications in various fields, including autonomous driving [1], action recognition [2], and virtual reality [3]. Many scene flow estimation methods based on various types of input data have recently been proposed, such as image sequence [4] [5], 3D point clouds [6] [7]. However, the acquisition of 3D point cloud data usually requires expensive sensor equipment, so the research in this paper focuses on the scene flow estimation methods whose input is image sequence.…”