Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

Abstract

We present a novel approach for unsupervised learning of depth and ego-motion from monocular video. Unsupervised learning removes the need for separate supervisory signals (depth or ego-motion ground truth, or multi-view video). Prior work in unsupervised depth learning uses pixel-wise or gradient-based losses, which only consider pixels in small local neighborhoods. Our main contribution is to explicitly consider the inferred 3D geometry of the whole scene, and enforce consistency of the estimated 3D point clouds and ego-motion across consecutive frames. This is a challenging task and is solved by a novel (approximate) backpropagation algorithm for aligning 3D structures.We combine this novel 3D-based loss with 2D losses based on photometric quality of frame reconstructions using estimated depth and ego-motion from adjacent frames. We also incorporate validity masks to avoid penalizing areas in which no useful information exists.We test our algorithm on the KITTI dataset and on a video dataset captured on an uncalibrated mobile phone camera. Our proposed approach consistently improves depth estimates on both datasets, and outperforms the stateof-the-art for both depth and ego-motion. Because we only require a simple video, learning depth and ego-motion on large and varied datasets becomes possible. We demonstrate this by training on the low quality uncalibrated video dataset and evaluating on KITTI, ranking among top performing prior methods which are trained on KITTI itself. 1

Spherical View Synthesis for Self-Supervised 360° Depth Estimation

“…However, learning to estimate depth purely from video breaks the static scene assumption and necessitates the use of an attention mechanism for foreground motion between consecutive frames. More recent iterations of this direction added scale normalization and removal of the separate pose estimation branch [40], 3D geometric constraints between the pre-dicted depths [23], epipolar constraints [29], additional feature reconstruction supervision [44], stereo matching constraints [43] or explicitly used two consecutive frames as input [28].…”

A Neural Network for Detailed Human Depth Estimation From a Single Image

“…Depth estimation from a single image has gained increasing attention in the computer vision community. Most works like [37,38,20,15,39,41,9,16] are proposed for indoor and outdoor scenes. We focus on depth estimation of humans, which allows us to build much stronger shape prior than these generic depth estimation methods.…”

Moving Indoor: Unsupervised Video Depth Learning in Challenging Environments

“…There are also some approaches proposed to get rid of the PoseNet. For instance, Mahjourian et al [32] used Iterative Closest Point (ICP) [2,4,36] to compute a transformation that minimizes point-to-point distances between corresponding points. Wang et al [42] used direct visual odometry (DVO) [40] to obtain camera pose from predicted depth and images.…”