Monocular Depth Estimation Based on Multi-Scale Depth Map Fusion

Abstract

Monocular depth estimation is a basic task in machine vision. In recent years, the performance of monocular depth estimation has been greatly improved. However, most depth estimation networks are based on a very deep network to extract features that lead to a large amount of information lost. The loss of object information is particularly serious in the encoding and decoding process. This information loss leads to the estimated depth maps lacking object structure detail and have non-clear edges. Especially in a complex indoor environment, which is our research focus in this paper, the consequences of this loss of information are particularly serious. To solve this problem, we propose a Dense feature fusion network that uses a feature pyramid to aggregate various scale features. Furthermore, to improve the fusion effectiveness of decoded object contour information and depth information, we propose an adaptive depth fusion module, which allows the fusion network to fuse various scale depth maps adaptively to increase object information in the predicted depth map. Unlike other work predicting depth maps relying on U-NET architecture, our depth map predicted by fusing multi-scale depth maps. These depth maps have their own characteristics. By fusing them, we can estimate depth maps that not only include accurate depth information but also have rich object contour and structure detail. Experiments indicate that the proposed model can predict depth maps with more object information than other prework, and our model also shows competitive accuracy. Furthermore, compared with other contemporary techniques, our method gets state-of-the-art in edge accuracy on the NYU Depth V2 dataset.

Monocular Depth Estimation with Sharp Boundary

“…The multi-scale feature fusion is an effective method to deal with low-level feature loss. The Fuse Feature Pyramid (FFP) is a usual method to aggregate different scales feature, such as Chen et al [17], and Yang et al [18], but these models have a lot of parameters due to sampling the features too many times during building the FFP. To reduce the parameters, SU and ST are designed in this work.…”

“…Based on this excellent work, Chen et al [17] proposed a Fused Feature Pyramid (FFP) and a residual pyramid to predict depth maps. Yang et al [18] built an FFP and used an ASFF (Adaptively Spatial Feature Fusion) structure [29] to fuse the different scale depth maps to keep the structure information. Although Chen et al [17] and Yang et al [18] showed a great performance, their model has a lot of parameters for building the FFP.…”

“…Yang et al [18] built an FFP and used an ASFF (Adaptively Spatial Feature Fusion) structure [29] to fuse the different scale depth maps to keep the structure information. Although Chen et al [17] and Yang et al [18] showed a great performance, their model has a lot of parameters for building the FFP. Based on this work, we propose a SU module to fuse the multi-scale feature, which learns the global scene information.…”

“…The ST not only changes the resolution of global information but also adaptively learn what features are needed in different decode scale and change the feature. With ST building a pyramid reduces a lot of model parameters than pre-works, such as Chen et al [17] and Yang et al [18]. The comparison results of the parameters of several models can be seen in the experiment section.…”

“…SUN RGB-D [20] is a scene understanding benchmark, which includes three datasets NYU-Depth V2 [19], B3DO [37], SUN3D [38]. To ensure the effectiveness of the comparison, we choose Hu et al [15], Chen et al [17], and Yang et al [18] as the comparison models. We use the same dataset [19] to train the model.…”