Semantic 3D occupancy mapping through efficient high order CRFs

Abstract

Semantic 3D mapping can be used for many applications such as robot navigation and virtual interaction. In recent years, there has been great progress in semantic segmentation and geometric 3D mapping. However, it is still challenging to combine these two tasks for accurate and largescale semantic mapping from images. In the paper, we propose an incremental and (near) real-time semantic mapping system. A 3D scrolling occupancy grid map is built to represent the world, which is memory and computationally efficient and bounded for large scale environments. We utilize the CNN segmentation as prior prediction and further optimize 3D grid labels through a novel CRF model. Superpixels are utilized to enforce smoothness and form robust P N high order potential. An efficient mean field inference is developed for the graph optimization. We evaluate our system on the KITTI dataset and improve the segmentation accuracy by 10% over existing systems.

Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping

“…Basic CRF models encourage label consistency for adjacent 3D elements, while higher-order dense CRFs can model longrange relationships within a region, such as grids in supervoxels [8] or grids corresponding to 2D superpixels [12], and further improve the mapping performance. With the advent of deep learning methods, recent work uses deep Convolutional Neural Networks (CNNs) for 2D image segmentation, and follows the same framework for building 3D semantic maps [9], [27]. However, CRF optimization postprocesses the inferred occupied grids, which does not change the principle of discrete semantic map inference.…”

“…In the context of semantic occupancy mapping, the query points are chosen to be the grid centroids. Thus, (9) can be used to recursively update the posterior parameters for each grid. We use a block to contain a number of grids according to the block depth, where each block is an octree of grids.…”

“…The input data of this dataset consists of stereo camera images; we first transform the stereo images into 3D point clouds using the provided camera calibration, then we use an image segmentation deep neural network [39] on the left image to obtain semantic measurements. For a fair comparison, we adopt the same data preprocessing methods as used by Yang et al [9]. We use ELAS [40] to generate depth maps from stereo image pairs, ORB-SLAM [41] to estimate 6DoF camera poses, and the deep network dilated CNN [39] for prior semantic label predictions.…”

“…3: Qualitative results on KITTI odometry sequence 05 test set [13]. From left to right the figures show 2D projected images from Yang et al [9], S-CSM and S-BKI, respectively. on labeled ground truth in left images.…”

“…1, besides possessing properties similar to an occupancy grid map, maintains for each cell a set of probabilities of semantic classes. These probabilities are often updated using a Bayes filter [9], [10], and then Conditional Random Fields (CRF) or Markov Random Fields (MRF) are subsequently applied to mitigate discontinuities and inconsistencies in the semantic map [7]- [9], [11], [12]. In principle, CRF models encourage label consistency among neighboring grids in super-voxels [8] or 2D superpixels [9], [12].…”