Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

Abstract

To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher-level entities during mapping, such as individual object instances. This work presents an approach to incrementally build volumetric objectcentric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an unsupervised geometric approach with instance-aware semantic predictions to detect both recognized scene elements as well as previously unseen objects. Next, a data association step tracks the predicted instances across the different frames. Finally, a map integration strategy fuses information about their 3D shape, location, and, if available, semantic class into a global volume. Evaluation on a publicly available dataset shows that the proposed approach for building instance-level semantic maps is competitive with state-of-the-art methods, while additionally able to discover objects of unseen categories. The system is further evaluated within a real-world robotic mapping setup, for which qualitative results highlight the online nature of the method. Code is available at https://github.com/ethz-asl/voxblox-plusplus.

Incremental Instance-Oriented 3D Semantic Mapping via RGB-D Cameras for Unknown Indoor Scene

“…Representation FQ FPS SemanticFusion [11] Dense Every 10 frames Under 8 Hz Hermans et al [34] Dense Every 6 frames 3 Hz PanopicFusion [44] Dense Every 10 frames 4.3 Hz Voxblox++ [16] Instance-oriented Every frame 1 Hz Pham et al [45] Instance-oriented Every frame 1 Hz Fusion++ [29] Instance-oriented progressively integrating predictions from partial observations into a global map but can learn from the entire 3D layout of the scene, they are not directly comparable with our work. Among the frameworks that study online, incremental instance-aware semantic mapping, we chose Grinvald et al [16] as a comparison. Because we relied on a Mask R-CNN model trained on the 80 Microsoft COCO [38] object classes to get the instance IDs, we evaluated the segmentation accuracy on the nine object categories that were common to the SceneNN dataset [21].…”

“…us, the approach in [11] does not provide any information about the geometry and relative placement of individual objects in the scene. A number of other works have addressed the task of detecting and segmenting individual semantically meaningful objects in 3D scenes without predefined shape templates [10,16,17,27,[29][30][31][32][33][34]. Runz et al [32] employed the object detector for the first step and then updated the class probabilities of each element consisting of the reconstructed 3D map.…”

“…As it has a huge time complexity, these methods suggested to only extract semantic information on a subset of the input frames; McCormac et al [29] utilized the same prediction model but aims at extending the SLAM system by means of object-level pose graph optimizations and relocalizations. [16,17] are similar to that, but they employ depth segmentation methods to segment 3D instances, which led them to take different approaches and reach different goals. [22] Proposes an object-oriented mapping system that combines a Single Shot MultiBox Detector (SSD) [6] with ORB-SLAM2 [24].…”

“…Our system achieved a speed of 10.8 Hz while performing all processing components on every input frame, thereby outperforming other similar Discrete Dynamics in Nature and Society systems in run-time tests. Compared to the process for utilizing the semantic information from the input image in conventional methods [16,29,32], the proposed system has substantially reduced the computational time by exploiting a voxel-based class probability update scheme. All systems were tested on the same sequences of the SceneNN dataset.…”

“…In the last step, we propose a voxel-based Bayesian update strategy towards incremental class update across different frames, which will be incorporated into the truncated signed distance function-(TSDF-) based reconstruction maps for the purpose of accelerating the computational efficiency and reducing time complexity. e major difference between our system and other works [10,16,17] is that we employ the projection relation between voxel and pixel directly to obtain instances semantic in the 3D map instead of using the combination between geometry segmentation on depth images and 2D instance segmentation methods. Doing so helps avoid oversegment with no computation increased.…”