FastDepth: Fast Monocular Depth Estimation on Embedded Systems

Abstract

Depth sensing is a critical function for robotic tasks such as localization, mapping and obstacle detection. There has been a significant and growing interest in depth estimation from a single RGB image, due to the relatively low cost and size of monocular cameras. However, state-of-the-art single-view depth estimation algorithms are based on fairly complex deep neural networks that are too slow for real-time inference on an embedded platform, for instance, mounted on a micro aerial vehicle. In this paper, we address the problem of fast depth estimation on embedded systems. We propose an efficient and lightweight encoder-decoder network architecture and apply network pruning to further reduce computational complexity and latency. In particular, we focus on the design of a low-latency decoder. Our methodology demonstrates that it is possible to achieve similar accuracy as prior work on depth estimation, but at inference speeds that are an order of magnitude faster. Our proposed network, FastDepth, runs at 178 fps on an NVIDIA Jetson TX2 GPU and at 27 fps when using only the TX2 CPU, with active power consumption under 10 W. FastDepth achieves close to state-of-the-art accuracy on the NYU Depth v2 dataset. To the best of the authors' knowledge, this paper demonstrates real-time monocular depth estimation using a deep neural network with the lowest latency and highest throughput on an embedded platform that can be carried by a micro aerial vehicle.1 This throughput is achieved with a batch size of one and 32-bit floating point precision. Throughput can be increased by using a larger batch size (at the cost of higher latency), and/or reducing bitwidths through quantization.2 Accuracy metrics are defined in Section IV-A.

Bypassing Depth Maps Transmission For Immersive Video Coding

“…Using a GPU-optimized implementation, they achieve to estimate around one HD depth map per second. The authors of [12] use several runtime optimization strategies on their deep convolutional encoder-decoder design, such as depthwise decomposition, network pruning and hardware-specific compilation. Their CNN-based monocular depth estimator achieves up to 178 fps on 224x224 resolution video, while maintaining state-of-the art accuracy.…”

Superb Monocular Depth Estimation Based on Transfer Learning and Surface Normal Guidance

“…The decoder layers of the RSN network were also designed based on the Densenet-121 model. Different from the CDE network, up-projection units [ 17 ] ( and ) were employed instead of bilinear interpolation for boosting the surface normal estimation process. The detailed structure of up-projection units is shown in the upper row of Figure 6 b.…”

“…The up-projection unit [ 17 ] was designed for embed platforms. As shown in Figure 10 , it achieves the lowest latency in this experiment.…”

“… Comparison of time consumption: Runtimes of different up-sampling methods including. 2× bilinear interpolation [ 6 ], up and down projection [ 48 ], and up-projection [ 17 ]. …”

“…As for the second content, previous authors [ 6 , 17 , 18 ] chose to propose a lighter-weight model-based encoder–decoder network, which produced more accurate depth estimation maps and reduced computational complexity. Wofk et al [ 17 ] demonstrated that a well-designed low latency network can maintain real-time depth estimation with economical computation costs. Alhashim et al [ 6 ] proved that a very simple transfer learning-based decoder robustly achieves high-resolution depth maps.…”