Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction

Abstract

Abstract-Our abilities in scene understanding, which allow us to perceive the 3D structure of our surroundings and intuitively recognise the objects we see, are things that we largely take for granted, but for robots, the task of understanding large scenes quickly remains extremely challenging. Recently, scene understanding approaches based on 3D reconstruction and semantic segmentation have become popular, but existing methods either do not scale, fail outdoors, provide only sparse reconstructions or are rather slow. In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time. We also present a 'semantic fusion' approach that allows us to handle dynamic objects more effectively than previous approaches. We demonstrate the effectiveness of our approach on the KITTI dataset, and provide qualitative and quantitative results showing high-quality dense reconstruction and labelling of a number of scenes.

Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping

“…To deal with noisy 2D predictions, 3D CRF optimization has been introduced as a refinement technique and it is widely used in 3D semantic mapping [7], [11], [26]. In [8], [12], [13], a higher-order dense CRF model is used to further optimize the semantic predictions for 3D elements.…”

“…1, besides possessing properties similar to an occupancy grid map, maintains for each cell a set of probabilities of semantic classes. These probabilities are often updated using a Bayes filter [9], [10], and then Conditional Random Fields (CRF) or Markov Random Fields (MRF) are subsequently applied to mitigate discontinuities and inconsistencies in the semantic map [7]- [9], [11], [12]. In principle, CRF models encourage label consistency among neighboring grids in super-voxels [8] or 2D superpixels [9], [12].…”

Visual-Based Semantic SLAM with Landmarks for Large-Scale Outdoor Environment

“…The method proposed in [29] was the early work utilizing stereo vision and classifying image to separate the traversable and nontraversable scenes with SVM. Furthermore, the algorithm described in [30] generated an efficient and accurate dense 3D reconstruction with associated semantic labels. Conditional Random Field (CRF) framework was applied to operate on stereo images to estimate labels and annotate the 3D volume.…”

Semantically Coherent Co-Segmentation and Reconstruction of Dynamic Scenes

“…This concept was extended to semantic segmentation and reconstruction to obtain additional information from the scene [24,56]. Methods were introduced to utilize appearance-based pixel categories and stereo cues in a joint framework for street scenes from a monocular camera [34,55,18]. These methods used CRF to perform simultaneous dense reconstruction and segmentation of street scenes captured from a moving camera.…”