Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

Abstract

Monocular depth estimation is a fundamental problem for various vision applications, and is therefore gaining increasing attention in the field of computer vision. Though a great improvement has been made thanks to the rapid progress of deep convolutional neural networks, depth estimation of the object at finer details remains an unsatisfactory issue, especially in complex scenes that has rich structure information. In this paper, we proposed a deep end-to-end learning framework with the combination of multi-scale convolutions and joint attention mechanisms to tackle this challenge. Specifically, we firstly elaborately designed a lightweight up-convolution to generate multi-scale feature maps. Then we introduced an attention-based residual block to aggregate different feature maps in joint channel and spatial dimension, which could enhance the discriminant ability of feature fusion at finer details. Furthermore, we explored an effective adaptive weight adjustment strategy for the loss function to further improve the performance, which adjusts the weight of each loss term during training without additional hyperparameters. The proposed framework was evaluated using challenging NYU Depth v2 and KITTI datasets. Experimental results demonstrated that the proposed approach is superior to most of the state-of-the-art methods. INDEX TERMS Monocular depth estimation, multi-scale convolutions, joint attention mechanisms, weight adjustment.

Deep learning‐based object identification with instance segmentation and pseudo‐LiDAR point cloud for work zone safety

“…The REL value is slightly greater than the value in Liu et al. (2020), and the reason is that partial pedestrian samples with occlusion cause the errors of detected bounding boxes. Our vision‐based model's advantage is to avoid relying on LiDAR and improve the model's generalization ability.…”

“…On the KITTI dataset, the absolute relative difference (REL) and root mean squared error (RMSE) for our model are 0.120 and 2.865, respectively. Table 7 compares our method with other methods (J. Jiang et al., 2019; Liu et al., 2020; Xue et al. 2020), and our method has the lowest RMSE value.…”

Multilevel Pyramid Network for Monocular Depth Estimation Based on Feature Refinement and Adaptive Fusion

“…The results of the experiments on the NYUv2 dataset are reported in Table 1, in which our method is compared with other depth recovery approaches [4,5,17,18,21,22,[45][46][47][48][49][50][51][52]. By analyzing Table 1, Lee et al [50] explored relative depth and achieved optimal values for metrics and δ < 1.253.…”

Monocular Depth Estimation with Joint Attention Feature Distillation and Wavelet-Based Loss Function

“…Jiang et al [ 27 ] proposed an adaptive weight allocation method based on a Gaussian model for their proposed hybrid loss function. Liu et al [ 28 ] proposed an effective adaptive weight adjustment strategy to adjust each loss term’s weight during training. Lee et al [ 29 ] proposed a loss rebalancing algorithm to initialize and rebalance weights for loss terms adaptively during training.…”

“…Several loss function terms are commonly combined to construct loss functions for predicting a better-quality depth. Various weight-setting methods for the loss function terms have been proposed to balance the training process [ 27 , 28 , 29 ], but how to enhance loss function effectiveness for fixed loss term combinations remains an open question.…”