Temporally Consistent Depth Prediction With Flow-Guided Memory Units

Abstract

Predicting depth from a monocular video sequence is an important task for autonomous driving. Although it has advanced considerably in the past few years, recent methods based on convolutional neural networks (CNNs) discard temporal coherence in the video sequence and estimate depth independently for each frame, which often leads to undesired inconsistent results over time. To address this problem, we propose to memorize temporal consistency in the video sequence, and leverage it for the task of depth prediction. To this end, we introduce a two-stream CNN with a flow-guided memory module, where each stream encodes visual and temporal features, respectively. The memory module, implemented using convolutional gated recurrent units (ConvGRUs), inputs visual and temporal features sequentially together with optical flow tailored to our task. It memorizes trajectories of individual features selectively and propagates spatial information over time, enforcing a long-term temporal consistency to prediction results. We evaluate our method on the KITTI benchmark dataset in terms of depth prediction accuracy, temporal consistency and runtime, and achieve a new state of the art. We also provide an extensive experimental analysis, clearly demonstrating the effectiveness of our approach to memorizing temporal consistency for depth prediction.

MobileXNet: An Efficient Convolutional Neural Network for Monocular Depth Estimation

“…The contributions of this study are summarized as follows: (1) We introduce a novel CNN architecture for achieving running efficient and accurate depth estimation from a single image; (2) We design a hybrid loss function; (3) We evaluate the proposed method on a dataset consisting of small sized synthetic images. In particular, we show that the size of the filters used in the input layer of CNNs has an influence on the performance of MDE when small sized images are processed; (4) We use Pareto Optimality to compare the error and running time over different methods, which has not been exploited in depth estimation; and (5) We demonstrate that the proposed method not only generates comparable accuracy to the state-of-the-art methods which use either extremely deep and complex architecture or post-processing but also runs much faster on a single less powerful GPU.…”

“…Autonomous vehicles have been extensively used in many applications, such as aerial surveillance, search and rescue. In order to safely operate in cluttered and unpredictable environments, these vehicles require a strong awareness of their operational surroundings, in particular, the ability to detect and avoid stationary or mobile obstacles [1], [2]. Depth estimation provides a geometry-independent paradigm in order to detect free, navigable space with the minimum safe distance.…”

Stad: Stable Video Depth Estimation

“…With advances in deep networks, Eigen et al [2] showed that monocular depth estimation with deep neural network yields significant gains in accuracy and speed compared to the traditional attempts with handcrafted features. Starting with the work of Eigen et al [2], many learning-based methods [1,3,4,5,6,7,8,9,10,11] have been studied and showed high accuracy in overall depth evaluation metrics. However, they still produce noisy and temporally flickering depth maps because they perform depth estimation on each frame independently or do not use temporal information correctly.…”

“…Several video-based depth estimation methods have been studied, and the core idea behind these methods is to utilize temporal information. Temporal consistency can be explicitly constrained [1,7,8,9,12,13] or implicitly constrained using the recurrent neural networks [7,10,11]. Eom et al [7] proposed the recurrent model with a flow-guided memory unit for consistent depth.…”

“…Temporal consistency can be explicitly constrained [1,7,8,9,12,13] or implicitly constrained using the recurrent neural networks [7,10,11]. Eom et al [7] proposed the recurrent model with a flow-guided memory unit for consistent depth. They used optical flow to align video frames and hidden states in a flow-guided memory module over time.…”