SLAM-OR: Simultaneous Localization, Mapping and Object Recognition Using Video Sensors Data in Open Environments from the Sparse Points Cloud

Abstract

In this paper, we propose a novel approach that enables simultaneous localization, mapping (SLAM) and objects recognition using visual sensors data in open environments that is capable to work on sparse data point clouds. In the proposed algorithm the ORB-SLAM uses the current and previous monocular visual sensors video frame to determine observer position and to determine a cloud of points that represent objects in the environment, while the deep neural network uses the current frame to detect and recognize objects (OR). In the next step, the sparse point cloud returned from the SLAM algorithm is compared with the area recognized by the OR network. Because each point from the 3D map has its counterpart in the current frame, therefore the filtration of points matching the area recognized by the OR algorithm is performed. The clustering algorithm determines areas in which points are densely distributed in order to detect spatial positions of objects detected by OR. Then by using principal component analysis (PCA)—based heuristic we estimate bounding boxes of detected objects. The image processing pipeline that uses sparse point clouds generated by SLAM in order to determine positions of objects recognized by deep neural network and mentioned PCA heuristic are main novelties of our solution. In contrary to state-of-the-art approaches, our algorithm does not require any additional calculations like generation of dense point clouds for objects positioning, which highly simplifies the task. We have evaluated our research on large benchmark dataset using various state-of-the-art OR architectures (YOLO, MobileNet, RetinaNet) and clustering algorithms (DBSCAN and OPTICS) obtaining promising results. Both our source codes and evaluation data sets are available for download, so our results can be easily reproduced.

RDSP-SLAM: Robust Object-Aware SLAM Based on Deep Shape Priors

“…Object image semantic cues based only on 2DBboxes do not suffer from latency problems, as the results can be obtained in real-time. As such, RDS-SLAM [10], EAO-SLAM [7], and SLAM-OR [9] adopt a lightweight object detection neural network to extract 2DBox detection on all frames or keyframes in realtime while simultaneously tracking the camera, representing objects as quadrics, cuboids and voxels, and creating a scene map.…”

“…Visual mapping approaches based on SLAM techniques, perform detection on visual data for increased inclusion of spatial semantic information in the map [4]. Improving the semantic information obtained during the mapping process has given rise to the object-level/aware/oriented SLAM research field, where individual objects in the scene are prioritized and thus (a) recognized in camera images, (b) associated with the created map (c) either used as landmarks for camera localization [5], (d) and/or represented as quadrics [6], [7], cuboids [7]- [9], voxels [10], or surfels/mesh [11]- [13].…”

Key-Region-Based UAV Visual Navigation

“…This approach demonstrated strong geolocalization with an accuracy of 3.4 m over km long flights in urban and rural environments. Another study combined SLAM with object recognition to map the 3D location of detected objects (Mazurek and Hachaj, 2021).…”

WPO-Net: Windowed Pose Optimization Network for Monocular Visual Odometry Estimation

“…The method of performing the continuous localization using cameras or visual-only sensors is known as visual odometry (VO). The applications of visual odometry vary widely from scene reconstruction [1], indoor localization [2], biomedical applications [3], and virtual and augmented reality [4] to self-driving vehicles [5].…”