Real-Time RGB-D Simultaneous Localization and Mapping Guided by Terrestrial LiDAR Point Cloud for Indoor 3-D Reconstruction and Camera Pose Estimation

Abstract

In recent years, low-cost and lightweight RGB and depth (RGB-D) sensors, such as Microsoft Kinect, have made available rich image and depth data, making them very popular in the field of simultaneous localization and mapping (SLAM), which has been increasingly used in robotics, self-driving vehicles, and augmented reality. The RGB-D SLAM constructs 3D environmental models of natural landscapes while simultaneously estimating camera poses. However, in highly variable illumination and motion blur environments, long-distance tracking can result in large cumulative errors and scale shifts. To address this problem in actual applications, in this study, we propose a novel multithreaded RGB-D SLAM framework that incorporates a highly accurate prior terrestrial Light Detection and Ranging (LiDAR) point cloud, which can mitigate cumulative errors and improve the system’s robustness in large-scale and challenging scenarios. First, we employed deep learning to achieve system automatic initialization and motion recovery when tracking is lost. Next, we used terrestrial LiDAR point cloud to obtain prior data of the landscape, and then we applied the point-to-surface inductively coupled plasma (ICP) iterative algorithm to realize accurate camera pose control from the previously obtained LiDAR point cloud data, and finally expanded its control range in the local map construction. Furthermore, an innovative double window segment-based map optimization method is proposed to ensure consistency, better real-time performance, and high accuracy of map construction. The proposed method was tested for long-distance tracking and closed-loop in two different large indoor scenarios. The experimental results indicated that the standard deviation of the 3D map construction is 10 cm in a mapping distance of 100 m, compared with the LiDAR ground truth. Further, the relative cumulative error of the camera in closed-loop experiments is 0.09%, which is twice less than that of the typical SLAM algorithm (3.4%). Therefore, the proposed method was demonstrated to be more robust than the ORB-SLAM2 algorithm in complex indoor environments.

Perspectives of RealSense and ZED Depth Sensors for Robotic Vision Applications

“…These are principal features related to all depth cameras since the lenses, aperture, and light significantly affect the image quality of any camera, not only the depth camera. These features enable high-quality depth images to be obtained, upon which promising robotic vision applications can be developed [41][42][43][44][45][46][47][48][49][50][51][52][53][54][55][56][57][58][59]. According to the Stereolabs [16], the main features of ZED cameras are listed in Table 2.…”

“…Up to now, 3D depth cameras have been very expensive, and their utilization has been complex and burdensome from a hardware standpoint. Nowadays, thanks to technical progress, the price of 3D depth sensors that can measure image depth is considerably affordable and their use much simpler [41][42][43][44][45][46][47][48][49][50][51][52][53][54][55][56][57][58][59].…”

Recognition of Manual Welding Positions from Depth Hole Image Remotely Sensed by RGB-D Camera

“…Many studies have been conducted on body tracking and motion analysis using depth images [20][21][22][23]. There are two methods for creating depth images: extracting features from two-dimensional images and inferring depth through learning [24][25][26][27] or shooting with a 3D depth camera [28][29][30]. The former method has disadvantages in that an additional process is required to extract and learn features of an image, it takes a lot of time, and the accuracy is low.…”

2D LiDAR and Camera Fusion Using Motion Cues for Indoor Layout Estimation

“…RGB-D sensors can provide both visual and depth information, which become popular in 3D reconstruction of cluttered and small-scale environments [10], [11]. However, the depth information is noisy and with limited field of view and depth range, so algorithms or even systems integrated with LiDARs [12], [13] have been developed to deal with the accumulated errors in tracking. For indoor layout estimation, focusing on polygonal structure modeling, fusion of 2D LiDAR and intensity camera can be a more efficient solution.…”