Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation

Abstract

Dense depth estimation is essential to sceneunderstanding for autonomous driving. However, recent selfsupervised approaches on monocular videos suffer from scaleinconsistency across long sequences. Utilizing data from the ubiquitously copresent global positioning systems (GPS), we tackle this challenge by proposing a dynamically-weighted GPS-to-Scale (g2s) loss to complement the appearance-based losses. We emphasize that the GPS is needed only during the multimodal training, and not at inference. The relative distance between frames captured through the GPS provides a scale signal that is independent of the camera setup and scene distribution, resulting in richer learned feature representations. Through extensive evaluation on multiple datasets, we demonstrate scale-consistent and -aware depth estimation during inference, improving the performance even when training with low-frequency GPS data.

Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics

“…Of note is that DynaDepth also achieves a nearly perfect absolute scale. In terms of scale-awareness, even our R18 version outperforms G2S R50 [3], which uses a heavier encoder. For better illustration, we also show the scaling ratio histograms with and without IMU in Fig.…”

“…However, the absolute scale is not guaranteed in these methods. Similar to DynaDepth, there exist methods that resort to other sensors than monocular camera, such as stereo camera that allows a scale-aware left-right consistency loss [13,14,46], and GPS that provides velocities to constrain the ego-motion network [15,3]. In comparison with these methods, using IMU is beneficial in that (1) IMU provides better generalizability since it does suffer from the visual domain gap, and (2) unlike GPS that cannot be used indoors and cameras that fail in texture-less, dynamic and illumination changing scenes, IMU is more robust to the environments.…”

“…In addition to the removal of static frames as proposed in [48], we discard images without the corresponding IMU records, leading to 38,102 image-and-IMU triplets for training and 4,238 for validation. WLOG, we use the image resolution 640x192 and cap the depth predictions at 80m, following the common practice in [14,18,15,3,38]. We compare DynaDepth with state-of-the-art monocular depth estimation methods in Table 1, which rescale the results using the ratio of the median depth between the ground-truth and the prediction.…”

Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics

“…Online Benchmark: We also measure the performance of MT-SfMLearner on the KITTI Online Benchmark for depth prediction 1 using the metrics from (Uhrig et al, 2017). We train on an image size of 1024 × 320, and add the G2S loss (Chawla et al, 2021) for obtaining predictions at metric scale. Results ordered by their rank are shown in Table 4.…”

“…However, supervised methods require extensive RGB-D ground truth collected from costly LiDARs or multi-camera rigs. Instead, self-supervised methods have increasingly utilized concepts of Structure from Motion (SfM) with known camera intrinsics to train monocular depth and ego-motion estimation networks simultaneously (Guizilini et al, 2020;Lyu et al, 2020;Chawla et al, 2021). While transformer ingredients such as attention have been utilized for self-supervised depth estimation (Johnston and Carneiro, 2020), most methods are nevertheless limited to the use of CNNs that have localized linear operations and lose feature resolution during downsampling to increase their limited receptive field (Yang et al, 2021).…”