Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks

Mousavian, Arsalan; Pirsiavash, Hamed; Košecká, Jana

doi:10.1109/3dv.2016.69

Cited by 136 publications

(121 citation statements)

References 24 publications

Supporting

Mentioning

119

Contrasting

Unclassified

Order By: Relevance

“…Ladicky et al [36] combine depth regression with semantic classification deploying a bag-of-visual-words model and a boosting algorithm. Mousavian et al [37] deploy a multi-scale CNN to estimate depth and used within a CRF to obtain semantic segmentation. Wang et al [38] use local and global CNNs to extract pixels and regions potential which are fed to a CRF.…”

Section: Related Workmentioning

confidence: 99%

Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation

Ramirez

Poggi

Tosi

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Depth estimation from a single image represents a very exciting challenge in computer vision. While other image-based depth sensing techniques leverage on the geometry between different viewpoints (e.g., stereo or structure from motion), the lack of these cues within a single image renders ill-posed the monocular depth estimation task. For inference, state-of-the-art encoder-decoder architectures for monocular depth estimation rely on effective feature representations learned at training time. For unsupervised training of these models, geometry has been effectively exploited by suitable images warping losses computed from views acquired by a stereo rig or a moving camera. In this paper, we make a further step forward showing that learning semantic information from images enables to improve effectively monocular depth estimation as well. In particular, by leveraging on semantically labeled images together with unsupervised signals gained by geometry through an image warping loss, we propose a deep learning approach aimed at joint semantic segmentation and depth estimation. Our overall learning framework is semi-supervised, as we deploy groundtruth data only in the semantic domain. At training time, our network learns a common feature representation for both tasks and a novel cross-task loss function is proposed. The experimental findings show how, jointly tackling depth prediction and semantic segmentation, allows to improve depth estimation accuracy. In particular, on the KITTI dataset our network outperforms state-of-the-art methods for monocular depth estimation.

show abstract

Section: Related Workmentioning

confidence: 99%

Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation

Ramirez

Poggi

Tosi

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…SSeg: The output of the segmenter defined by Mousavian et al [29] that is trained on NYU V2 dataset [30]. The resulting representation is a H × W × C sseg mask stack where C sseg is the number of categories in the NYU V2 dataset.…”

Section: B Visual Representationsmentioning

confidence: 99%

“…Then the gap between real and simulated data is related to the quality of the segmentation and detector. This setup is particularly timely as research on object detection [28], semantic segmentation [35] and depth estimation [29] has been propelled by deep learning methods with a variety of high performing models available. We show empirically that by using simulation with these representations we achieve substantial improvement without any domain adaptation.…”

Section: B Visual Representationsmentioning

confidence: 99%

Visual Representations for Semantic Target Driven Navigation

Mousavian

Toshev

Fišer

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

194

127

View full text Add to dashboard Cite

What is a good visual representation for navigation? We study this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a previously unseen environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues.We propose to use semantic segmentation and detection masks as observations obtained by state-of-the-art computer vision algorithms and use a deep network to learn the navigation policy. The availability of equitable representations in simulated environments enables joint training using real and simulated data and alleviates the need for domain adaptation or domain randomization commonly used to tackle the sim-toreal transfer of the learned policies. Both the representation and the navigation policy can be readily applied to real nonsynthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach successfully gets to the target in 54% of the cases in unexplored environments, compared to 46% for a non-learning based approach, and 28% for a learning-based baseline.

show abstract

“…Several researchers have recently attempted this demanding challenge, i.e., building a model that can simultaneously learn multiple tasks with different outputs. Mousavian et al [310] undertook joint people detection in tandem with re-identification, while Van Ranst et al [311] tackled image segmentation with depth estimation. However, more exploration and investigation to overcome this challenge is needed.…”

Section: G Other Challengesmentioning

confidence: 99%

Survey on Multi-Output Learning

Shi

Tsang

et al. 2019

IEEE Trans. Neural Netw. Learning Syst.

164

View full text Add to dashboard Cite

The aim of multi-output learning is to simultaneously predict multiple outputs given an input. It is an important learning problem for decision-making, since making decisions in the real world often involves multiple complex factors and criteria. In recent times, an increasing number of research studies have focused on ways to predict multiple outputs at once. Such efforts have transpired in different forms according to the particular multi-output learning problem under study. Classic cases of multi-output learning include multi-label learning, multidimensional learning, multi-target regression and others. From our survey of the topic, we were struck by a lack in studies that generalize the different forms of multi-output learning into a common framework. This paper fills that gap with a comprehensive review and analysis of the multi-output learning paradigm. In particular, we characterize the 4 Vs of multi-output learning, i.e., volume, velocity, variety, and veracity, and the ways in which the 4 Vs both benefit and bring challenges to multioutput learning by taking inspiration from big data. We analyze the life cycle of output labeling, present the main mathematical definitions of multi-output learning, and examine the field's key challenges and corresponding solutions as found in the literature. Several model evaluation metrics and popular data repositories are also discussed. Last but not least, we highlight some emerging challenges with multi-output learning from the perspective of the 4 Vs as potential research directions worthy of further studies.

show abstract

Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks

Cited by 136 publications

References 24 publications

Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation

Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation

Visual Representations for Semantic Target Driven Navigation

Survey on Multi-Output Learning

Contact Info

Product

Resources

About