Don't Forget The Past: Recurrent Depth Estimation from Monocular Video

Abstract

Autonomous cars need continuously updated depth information. Thus far, the depth is mostly estimated independently for a single frame at a time, even if the method starts from video input. Our method produces a time series of depth maps, which makes it an ideal candidate for online learning approaches. In particular, we put three different types of depth estimation (supervised depth prediction, self-supervised depth prediction, and self-supervised depth completion) into a common framework. We integrate the corresponding networks with a convolutional LSTM such that the spatiotemporal structures of depth across frames can be exploited to yield a more accurate depth estimation. Our method is flexible. It can be applied to monocular videos only or be combined with different types of sparse depth patterns. We carefully study the architecture of the recurrent network and its training strategy. We are first to successfully exploit recurrent networks for real-time self-supervised monocular depth estimation and completion. Extensive experiments show that our recurrent method outperforms its image-based counterpart consistently and significantly in both self-supervised scenarios. It also outperforms previous depth estimation methods of the three popular groups. The code is available at: www.github.com/wvangansbeke/

DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion

“…This allows for modelling the relations between the frames, even without knowledge of the camera poses. Several single image depth prediction methods [36,47,54] have shown that ConvLSTM cells can improve the temporal consistency of the depth predictions.…”

“…In [36], input frames and sparse depth cues are encoded together, then temporal relations between encodings are modeled through ConvLSTM. Finally, the depth predictions are output by a decoder.…”

“…Apple ARKit and Android ARCore) [19] or mixed reality headsets such as Microsoft HoloLens. The presence of camera poses enables the computation of triangulation-based metric reconstructions, as opposed to the popular learning-based single image depth prediction methods [3,11,13,28,52] that lately have been extended to video [36,47,49,54]. Finally, the real-time aspect of the aforementioned applications and the potential of an on-device solution, imply targeting an online multi-view stereo (MVS) system that is efficient and lightweight, both in terms of memory and computation load.…”

“…In this work, we present a framework that can extend many existing MVS methods, such that when processing video streams, partial scene geometry information from the past contributes to the prediction at the current time step, leading to improved and consistent depth outputs.We use convolutional long short-term memory (ConvLSTM) [42] and a hidden state propagation scheme to achieve such information flow in the latent space. Our approach is influenced by [19] where latent cost volume encodings are weakly coupled through a Gaussian Process, and by [36] where latent encodings of image features and sparse depth cues are fused through ConvLSTM to achieve temporally consistent depth predictions. However, we leverage geometry and explicitly account for the implications of perspective projection when propagating the latent encodings.…”

Variational Monocular Depth Estimation for Reliability Prediction

“…In recent years, research on monocular depth estimation by self-supervised learning has become an important field in deep learning [4,38,14,27,45,46,44,47,20,48,41]. The developments in spatial transformer modules have further enhanced research on monocular depth estimation [19].…”