Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework Using Visual and Depth Cues

Abstract

This paper addresses the problem of building augmented metric representations of scenes with semantic information from RGB-D images. We propose a complete framework to create an enhanced map representation of the environment with object-level information to be used in several applications such as human-robot interaction, assistive robotics, visual navigation, or in manipulation tasks. Our formulation leverages a CNN-based object detector (Yolo) with a 3D model-based segmentation technique to perform instance semantic segmentation, and to localize, identify, and track different classes of objects in the scene. The tracking and positioning of semantic classes is done with a dictionary of Kalman filters in order to combine sensor measurements over time and then providing more accurate maps. The formulation is designed to identify and to disregard dynamic objects in order to obtain a mediumterm invariant map representation. The proposed method was evaluated with collected and publicly available RGB-D data sequences acquired in different indoor scenes. Experimental results show the potential of the technique to produce augmented semantic maps containing several objects (notably doors). We also provide to the community a dataset composed of annotated object classes (doors, fire extinguishers, benches, water fountains) and their positioning, as well as the source code as ROS packages. 1 1 Preprint paper version to appear at Journal of Intelligent & Robotic Systems, available online at: https://doi.

An Object-oriented Navigation Strategy for Service Robots Leveraging Semantic Information

“…After the cost-association matrix is computed, the association between new observations (i.e list C) and previous observations (i.e dictionary P) is determined using the Hungarian algorithm [33] as prescribed in [22]. If the distance corresponding to the association is less than the threshold value α (D k,l < α), it is assumed to correspond to the previously seen object with which it is associated to.…”

“…Martins et al demonstrated semantic mapping in the real world with off-beat object classes like fire hydrants. However, their approach does not cover 'Go to Object' navigation [22]. Using semantics for path planning is largely an unexplored area of research.…”

Monocular camera and laser based semantic mapping system with temporal-spatial data association for indoor mobile robots

“…The experimental results verify the effectiveness in solving a novel RGB-D object detection and recognition application with limited human annotations. Renato et al [16] proposed a learning-based object-augmented semantic mapping system, combining the environment structure and object semantics using visual and depth cues. This approach models the detected semantic classes with a constant Kalman filter module to track and update the most probable object position.…”

An Online Semantic Mapping System for Extending and Enhancing Visual SLAM

“…Li et al (2019) derive 3D bounding cuboids of objects from a sequence of 2D detections for relocalization from large viewpoint changes. Martins et al (2020) extend robotic maps with semantic 3D shape priors that have similar appearance to projected object detections. On the contrary, Bowman et al (2017) and Doherty et al (2020) propose to directly incorporate semantic constraints into an probabilistic optimization framework.…”