Latent 3D Volume for Joint Depth Estimation and Semantic Segmentation from a Single Image

Abstract

This paper proposes a novel 3D representation, namely, a latent 3D volume, for joint depth estimation and semantic segmentation. Most previous studies encoded an input scene (typically given as a 2D image) into a set of feature vectors arranged over a 2D plane. However, considering the real world is three-dimensional, this 2D arrangement reduces one dimension and may limit the capacity of feature representation. In contrast, we examine the idea of arranging the feature vectors in 3D space rather than in a 2D plane. We refer to this 3D volumetric arrangement as a latent 3D volume. We will show that the latent 3D volume is beneficial to the tasks of depth estimation and semantic segmentation because these tasks require an understanding of the 3D structure of the scene. Our network first constructs an initial 3D volume using image features and then generates latent 3D volume by passing the initial 3D volume through several 3D convolutional layers. We apply depth regression and semantic segmentation by projecting the latent 3D volume onto a 2D plane. The evaluation results show that our method outperforms previous approaches on the NYU Depth v2 dataset.

Monocular Depth Estimation with Joint Attention Feature Distillation and Wavelet-Based Loss Function

“…Several recent multitask learning methods [ 35 , 36 , 37 , 38 , 39 , 40 ] have been successfully introduced for MDE by estimating depth maps with other information, such as semantic segmentation labels, surface normals, super pixels, etc. For example, Eigen and Fergus [ 35 ] combined semantic segmentation, surface normal, and depth estimation cues to build a single DCNN.…”

“…This single architecture simplifies implementing a system that requires multiple prediction tasks. Ito et al [ 36 ] proposed a 3D representation for semantic segmentation and depth estimation from a single image. Lin et al [ 37 ] proposed a hybrid DCNN to integrate semantic segmentation and depth estimation into a unified framework.…”