Multi-Robot Distributed Semantic Mapping in Unfamiliar Environments through Online Matching of Learned Representations

Abstract

We present a solution to multi-robot distributed semantic mapping of novel and unfamiliar environments. Most state-of-the-art semantic mapping systems are based on supervised learning algorithms that cannot classify novel observations online. While unsupervised learning algorithms can invent labels for novel observations, approaches to detect when multiple robots have independently developed their own labels for the same new class are prone to erroneous or inconsistent matches. These issues worsen as the number of robots in the system increases and prevent fusing the local maps produced by each robot into a consistent global map, which is crucial for cooperative planning and joint mission summarization. Our proposed solution overcomes these obstacles by having each robot learn an unsupervised semantic scene model online and use a multiway matching algorithm to identify consistent sets of matches between learned semantic labels belonging to different robots. Compared to the state of the art, the proposed solution produces 20-60% higher quality global maps that do not degrade even as many more local maps are fused. OPEN SOURCE SOFTWAREAn implementation of this solution is contributed in the "Sunshine" 3D semantic mapping ROS package, a general purpose single-and multi-robot semantic mapping system: https://gitlab.com/warplab/ros/sunshine.

Dynamic Semantic Occupancy Mapping Using 3D Scene Flow and Closed-Form Bayesian Inference

“…Other works differ in the deep neural network used (e.g., recurrent neural networks on consecutive frames [18], [19], 3D CNN for point clouds [20]), the map representation employed (point-cloud maps [7], [19] and voxel-based maps [9], [18], [21]), or the type of semantics (instancelevel [22], object-level [7], [23]- [25] and place-level [26]). More recently, distributed semantic mapping for multirobots [27], [28] and 3D scene graphs [29] are also trending research topics.…”

Dynamic Semantic Occupancy Mapping using 3D Scene Flow and Closed-Form Bayesian Inference

“…Other works differ in the deep neural network used (e.g., recurrent neural networks on consecutive frames [60], [9], 3D CNN for point clouds [16]), the map representation employed (point-cloud maps [52], [9] and voxel-based maps [60], [63], [33]), or the type of semantics (instance-level [19], object-level [52], [66], [45], [68] and place-level [51]). More recently, distributed semantic mapping for multi-robots [65], [26] and 3D scene graphs [44] are also trending research topics.…”