Joint Semantic Segmentation and 3D Reconstruction from Monocular Video

Abstract

Abstract. We present an approach for joint inference of 3D scene structure and semantic labeling for monocular video. Starting with monocular image stream, our framework produces a 3D volumetric semantic + occupancy map, which is much more useful than a series of 2D semantic label images or a sparse point cloud produced by traditional semantic segmentation and Structure from Motion(SfM) pipelines respectively. We derive a Conditional Random Field (CRF) model defined in the 3D space, that jointly infers the semantic category and occupancy for each voxel. Such a joint inference in the 3D CRF paves the way for more informed priors and constraints, which is otherwise not possible if solved separately in their traditional frameworks. We make use of class specific semantic cues that constrain the 3D structure in areas, where multiview constraints are weak. Our model comprises of higher order factors, which helps when the depth is unobservable. We also make use of class specific semantic cues to reduce either the degree of such higher order factors, or to approximately model them with unaries if possible. We demonstrate improved 3D structure and temporally consistent semantic segmentation for difficult, large scale, forward moving monocular image sequences. Fig. 1. Overview of our system. From monocular image sequence, we first obtain 2D semantic segmentation, sparse 3D reconstruction and camera poses. We then build a volumetric 3D map which depicts both 3D structure and semantic labels.

Semantic Segmentation via Multi-task, Multi-domain Learning

“…These methods have also been combined with deep networks [2,20]. For the 7 subsets of the KITTI dataset used in this paper [9,13,14,18,19,22,25], deep learning has never been used to tackle the semantic segmentation step. For example, [14] shows how to jointly classify pixels and predict their depth using a multi-class decision stumps-based boosted classifier.…”

“…For example, in the KITTI dataset (see Fig. 2 where all labels are reported) the class Tree of the dataset from He et al [9] is correlated with the class Vegetation from the dataset labeled by Kundu et al [13]. A plain softmax, optimizing the probability of the Tree class will implicitly penalize the probability of Vegetation, which is not a desired effect.…”

“…Most datasets for scene parsing contain only several hundreds of images, some of them only several dozen [6,15,9,13,14,18,19,22,25]. Additionally, combining these datasets is a non-trivial task as target classes are often tailored to a custom application.…”

“…One good example of such label-set diversity can be found within the KITTI Vision benchmark [4] which contains outdoor scene videos. Many research teams work on this dataset since its release in 2013, tackling computer vision tasks such as visual odometry, 3D object detection and 3D tracking [9,13,14,18,19,22,25]. To tackle these tasks, several research teams have labeled parts of the original dataset, independently from the other teams and often for different goals (among the works listed above, semantic segmentation is the final goal only for [14] and [25]).…”

Building Scene Models by Completing and Hallucinating Depth and Semantics

“…While impressive results can be achieved with multi-view and videobased approaches [1][2][3][4], the progress of depth sensors and their decreasing prices make them an attractive alternative, able to capture 3D in a single shot [5]. Unfortunately, even the best depth sensors still provide imperfect measurements.…”