Semantic segmentation with heterogeneous sensor coverages

Abstract

Fig. 1: Our novel semantic parsing approach can seamlessly integrate evidence from multiple sensors with overlapping but possibly different fields of view and account for missing data, while predicting semantic labels over the spatial union of sensors coverages. The semantic segmentation is formulated on a graph, in a manner which depends on sensing modality. First row: (a) over-segmentation on the image; (b) graph induced by superpixels; (c) 3D point cloud re-projected on the image with a tree graph structure computed in 3D, and (d) the full graph as proposed here for full scene understanding. In the second row is the semantic segmentation (a) ground truth and results of (b) using the image graph and only visual information; (c) using the 3D graph and visual and 3D information, and finally (d) the result from using a graph for full coverage and all the information. Note the best semantic segmentation achieved over the union of the spatial coverage of both sensors. Color code: ∎ground, ∎objects, ∎building and ∎vegetation.Abstract-We propose a new approach to semantic parsing, which can seamlessly integrate evidence from multiple sensors with overlapping but possibly different fields of view (FOV), account for missing data and predict semantic labels over the spatial union of sensors coverages. The existing approaches typically carry out semantic segmentation using only one modality, incorrectly interpolate measurements of other modalities or at best assign semantic labels only to the spatial intersection of coverages of different sensors. In this work we remedy these problems by proposing an effective and efficient strategy for inducing the graph structure of Conditional Random Field used for inference and a novel method for computing the sensor domain dependent potentials. We focus on RGB cameras and 3D data from lasers or depth sensors. The proposed approach achieves superior performance, compared to state of the art and obtains labels for the union of spatial coverages of both sensors, while effectively using appearance or 3D cues when they are available. The efficiency of the approach is amenable to realtime implementation. We quantitatively validate our proposal in two publicly available datasets from indoors and outdoors real environments. The obtained semantic understanding of the acquired sensory information can enable higher level tasks for autonomous mobile robots and facilitate semantic mapping of the environments.

Sensor fusion for semantic segmentation of urban scenes

“…The proposed algorithm is evaluated on the publicly available KITTI dataset [1] [2], augmented with additional pixel and point-wise semantic labels for building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence regions. A per-pixel accuracy of 89.3% and average class accuracy of 65.4% is achieved, well above current state-of-the-art [3]. …”

“…Feature vectors of the low-level segments are augmented with features computed on the corresponding segments from higherlevel segmentations. This is different from previous work [3], which uses localized features computed on a single scale only. Our approach precludes the need to train an expanding number of classifiers [20] or rely on complex graphical model machinery [17], but allows us to effectively integrate information from multiple scales.…”

“…In contrast to previous work [3] [20] [17], we explicitly take into account information from multiple scales up front by first performing segmentation on multiple scales and performing inference on the fine segmentation scale. Feature vectors of the low-level segments are augmented with features computed on the corresponding segments from higherlevel segmentations.…”

“…Several previous studies in semantic segmentation of urban 3D data have been recently performed [21] [22] [3]. Sengupta et al [21] and He et al [22] first perform semantic segmentation in the image modality.…”

“…Both methods disregard point cloud features. Cadena and Košecká [3] propose a pipeline to solve the semantic segmentation problem using a CRF framework, with an emphasis on fast run-time. The method first oversegments the image and extracts simple low-dimensional features.…”