Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation

Chen, Po‐Yi; Liu, Alexander H.; Liu, Yen‐Cheng; Wang, Yu-Chiang Frank

doi:10.1109/cvpr.2019.00273

Cited by 231 publications

(172 citation statements)

References 20 publications

Supporting

Mentioning

161

Contrasting

Order By: Relevance

“…Monocular depth estimation is a fundamental but challenging task in computer vision, the goal of which is to predict a dense depth map from a given image. The technical progress in this area can be applied to widespread applications, such as scene understanding [1], [2], action recognition [3], 3D reconstruction [4], robotics [5], [6], etc. However, it is still a very challenging topic since one image may correspond to several real scenes and there are no other available clues, e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

et al. 2020

View full text Add to dashboard Cite

Monocular depth estimation is a fundamental problem for various vision applications, and is therefore gaining increasing attention in the field of computer vision. Though a great improvement has been made thanks to the rapid progress of deep convolutional neural networks, depth estimation of the object at finer details remains an unsatisfactory issue, especially in complex scenes that has rich structure information. In this paper, we proposed a deep end-to-end learning framework with the combination of multi-scale convolutions and joint attention mechanisms to tackle this challenge. Specifically, we firstly elaborately designed a lightweight up-convolution to generate multi-scale feature maps. Then we introduced an attention-based residual block to aggregate different feature maps in joint channel and spatial dimension, which could enhance the discriminant ability of feature fusion at finer details. Furthermore, we explored an effective adaptive weight adjustment strategy for the loss function to further improve the performance, which adjusts the weight of each loss term during training without additional hyperparameters. The proposed framework was evaluated using challenging NYU Depth v2 and KITTI datasets. Experimental results demonstrated that the proposed approach is superior to most of the state-of-the-art methods. INDEX TERMS Monocular depth estimation, multi-scale convolutions, joint attention mechanisms, weight adjustment.

show abstract

Section: Introductionmentioning

confidence: 99%

“…represent the horizontal and vertical convolutional Sobel operator, which calculate the gradient information and are sensitive to the shift of edges in x and y directions 2. …”

mentioning

confidence: 99%

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

et al. 2020

View full text Add to dashboard Cite

show abstract

“…(11)). Similarly, Chen et al [74] also leveraged semantic segmentation to improve the monocular depth estimation. In ref.…”

Section: Semi-supervised Monocular Depth Estimationmentioning

confidence: 99%

Monocular depth estimation based on deep learning: An overview

Zhao¹,

Sun²,

Zhang³

et al. 2020

Sci. China Technol. Sci.

203

View full text Add to dashboard Cite

Depth information is important for autonomous systems to perceive environments and estimate their own state. Traditional depth estimation methods, like structure from motion and stereo vision matching, are built on feature correspondences of multiple viewpoints. Meanwhile, the predicted depth maps are sparse. Inferring depth information from a single image (monocular depth estimation) is an ill-posed problem. With the rapid development of deep neural networks, monocular depth estimation based on deep learning has been widely studied recently and achieved promising performance in accuracy. Meanwhile, dense depth maps are estimated from single images by deep neural networks in an end-to-end manner. In order to improve the accuracy of depth estimation, different kinds of network frameworks, loss functions and training strategies are proposed subsequently. Therefore, we survey the current monocular depth estimation methods based on deep learning in this review. Initially, we conclude several widely used datasets and evaluation indicators in deep learning-based depth estimation. Furthermore, we review some representative existing methods according to different training manners: supervised, unsupervised and semi-supervised. Finally, we discuss the challenges and provide some ideas for future researches in monocular depth estimation.

show abstract

“…Multi-task learning with the goal of learning multiple tasks simultaneously [ 40 ] is nowadays based on CNNs [ 13 , 15 , 16 , 17 , 18 , 19 , 20 , 41 , 42 , 43 , 44 , 44 ]. Eigen and Fergus proposed a network structure that can estimate the depth, semantic labels, and the surface orientation of a scene [ 13 ].…”

Section: Related Workmentioning

confidence: 99%

Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image

Ito

Kaneko

Kim

2020

Sensors

View full text Add to dashboard Cite

This paper proposes a novel 3D representation, namely, a latent 3D volume, for joint depth estimation and semantic segmentation. Most previous studies encoded an input scene (typically given as a 2D image) into a set of feature vectors arranged over a 2D plane. However, considering the real world is three-dimensional, this 2D arrangement reduces one dimension and may limit the capacity of feature representation. In contrast, we examine the idea of arranging the feature vectors in 3D space rather than in a 2D plane. We refer to this 3D volumetric arrangement as a latent 3D volume. We will show that the latent 3D volume is beneficial to the tasks of depth estimation and semantic segmentation because these tasks require an understanding of the 3D structure of the scene. Our network first constructs an initial 3D volume using image features and then generates latent 3D volume by passing the initial 3D volume through several 3D convolutional layers. We apply depth regression and semantic segmentation by projecting the latent 3D volume onto a 2D plane. The evaluation results show that our method outperforms previous approaches on the NYU Depth v2 dataset.

show abstract

Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation

Cited by 231 publications

References 20 publications

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

Joint Attention Mechanisms for Monocular Depth Estimation With Multi-Scale Convolutions and Adaptive Weight Adjustment

Monocular depth estimation based on deep learning: An overview

Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image

Contact Info

Product

Resources

About