A novel vSLAM framework with unsupervised semantic segmentation based on adversarial transfer learning

Abstract

Significant progress has been made in the field of visual Simultaneous Localization and Mapping (vSLAM) systems. However, the localization accuracy of vSLAM can be significantly reduced in dynamic applications with mobile robots or passengers. In this paper, a novel semantic SLAM framework in dynamic environments is proposed to improve the localization accuracy. We incorporate a semantic segmentation model into the Oriented FAST and Rotated BRIEF-SLAM2 (ORB-SLAM2) system to filter out dynamic feature points, but we encounter one main challenge, i.e. the performance of a segmentation network well-trained with labeled datasets may decrease seriously in a real application without any labeled data due to the inconsistency between the source domain and the target domain. Therefore, we proposed an unsupervised semantic segmentation model with a Residual Neural Network (ResNet) structure, which is trained by the adversarial transfer learning method in the multi-level feature spaces. This work may be the first to perform multi-level feature space adversarial transfer learning for the semantic SLAM task in dynamic environments. In order to evaluate our method, images of indoor scenes from three datasets are used as the source domain, and the dynamic sequences of the TUM dataset are used as the target domain. The extensive experimental results show favorable performance against the state-of-the-art methods in terms of the absolute trajectory accuracy and image semantic segmentation quality.

Deep Learning for Visual SLAM: The State-of-the-Art and Future Trends

“…Recently, three ways to develop deep learning-based VSLAM software components encompassing auxiliary modules, original deep learning modules, and end-to-end deep neural networks have been identified with different degrees of implementation. A way to develop auxiliary deep-based modules introduces most of the published studies including feature extraction [48][49][50], semantic segmentation [51][52][53][54][55][56][57][58][59][60], pose estimation [8,45,46,[61][62][63], map construction [3,[64][65][66], and loop closure [67][68][69][70]. It should be noted that deep neural networks extract low-level features from images by converting them to high-level featureslayer by layer.…”

“…Jin et al [57] proposed a semantic SLAM framework with Unsupervised Semantic Segmentation (USS-SLAM) in dynamic environments. The USS-SLAM framework ran four threads in parallel: tracking thread, local mapping thread, loop closing thread, and semantic map generation thread.…”

ADM-SLAM: Accurate and Fast Dynamic Visual SLAM with Adaptive Feature Point Extraction, Deeplabv3pro, and Multi-View Geometry

“…In the research field of intelligent robotics and autonomous navigation systems, Simultaneous Localization and Mapping (SLAM) technology plays a crucial role [ 1 , 2 ]. Notably, the work of M.W.…”

A real-time semantic visual SLAM for dynamic environment based on deep learning and dynamic probabilistic propagation

“…However, the actual scene inevitably contains moving objects or objects with moving properties. With the development of deep learning (DL), semantic SLAM for dynamic scenes has been widely studied in the literature [4][5][6][7], which mainly uses DL methods to detect moving objects in highly dynamic environments. However, most existing semantic SLAM methods for dynamic scenes still suffer from real-time performance issues.…”

“…However, most DL models used in the existing semantic SLAM methods require high memory consumption and computational cost. For example, the Mask R-CNN model [8] used in DynaSLAM [4] and OFM-SLAM [5], the Deeplab-V2 model [9] used in USS-SLAM [6], and the PSPNet-50 model [10] used in PSPNet-SLAM [7]. Some semantic SLAM methods [11][12][13] choose lightweight DL models, such as the SegNet model [14].…”