Deep Monocular Depth Estimation via Integration of Global and Local Predictions

Abstract

Recent works on machine learning have greatly advanced the accuracy of single image depth estimation. However, the resulting depth images are still over-smoothed and perceptually unsatisfying. This paper casts depth prediction from single image as a parametric learning problem. Specifically, we propose a deep variational model that effectively integrates heterogeneous predictions from two convolutional neural networks (CNNs), named global and local networks. They have contrasting network architecture and are designed to capture depth information with complementary attributes. These intermediate outputs are then combined in the integration network based on the variational framework. By unrolling the optimization steps of Split Bregman (SB) iterations in the integration network, our model can be trained in an end-to-end manner. This enables one to simultaneously learn an efficient parameterization of the CNNs and hyper-parameter in the variational method. Finally, we offer a new dataset of 0.22 million RGB-D images captured by Microsoft Kinect v2. Our model generates realistic and discontinuity-preserving depth prediction without involving any low-level segmentation or superpixels. Intensive experiments demonstrate the superiority of the proposed method in a range of RGB-D benchmarks including both indoor and outdoor scenarios.

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

“…Various datasets have been proposed that are suitable for monocular depth estimation, i.e. they consist of RGB images with corresponding depth annotation of some form [3], [11], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [40], [44], [45], [46], [47], [48]. Datasets differ in captured environments and objects (indoor/outdoor scenes, dynamic objects), type of depth annotation (sparse/dense, absolute/relative depth), accuracy (laser, time-of-flight, SfM, stereo, human annotation, synthetic data), image quality and camera settings, as well as dataset size.…”

“…We thus recreate the ground truth according to the procedure outlined by the original authors. DIML Indoor [31] (DL) is an RGB-D dataset of predominantly static indoor scenes, captured with a Kinect v2. Test datasets.…”

Soft Regression of Monocular Depth Using Scale-Semantic Exchange Network

“…The work of [22] is a representative of beginning using deep learning to solve monocular depth estimation. The work of [24] integrates heterogeneous predictions from global and local networks. Andrea Pilzer et.al [34] train a student network to predict a disparity map and a backward cycle network for generating image to re-synthesize back the input image.…”

“…Fu et al [13] apply dilated convolution for multi-scale receptive fields and develop a full-image encoder for global image properties. Kim et al [24] integrate heterogeneous predictions from global and local networks. These methods are difficult to maintain high resolution prediction.…”

Automatic Depth Estimation from Single 2D Image via Transfer Learning Approach

“…However, high computational resources and the large amount of data is required for training of deep learning model from scratch. In the literature, some of the authors proposed the dataaugmentation technique to enlarge the dataset for the suitable training of deep learning model [13], [14]. This is not a good approach as it increases training computation and may not be suitable for the real-time scenario.…”