Visual-Based Semantic SLAM with Landmarks for Large-Scale Outdoor Environment

Zhao, Zirui; Mao, Yijun; Ding, Yan; Ren, Pengju; Zheng, Nanning

doi:10.1109/cchi.2019.8901910

Cited by 26 publications

(13 citation statements)

References 33 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2020, Zhao et al [227] of Xi 'an Jiaotong University proposed a landmark visual semantic SLAM system for a large-scale outdoor environment. Its core is to combine a 3D point cloud in ORB-SLAM with semantic segmentation information in the convolutional neural network model PSPNET-101.…”

Section: Semantic With Locationmentioning

confidence: 99%

An Overview on Visual SLAM: From Tradition to Semantic

Chen

Shang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

Visual SLAM (VSLAM) has been developing rapidly due to its advantages of low-cost sensors, the easy fusion of other sensors, and richer environmental information. Traditional visionbased SLAM research has made many achievements, but it may fail to achieve wished results in challenging environments. Deep learning has promoted the development of computer vision, and the combination of deep learning and SLAM has attracted more and more attention. Semantic information, as high-level environmental information, can enable robots to better understand the surrounding environment. This paper introduces the development of VSLAM technology from two aspects: traditional VSLAM and semantic VSLAM combined with deep learning. For traditional VSLAM, we summarize the advantages and disadvantages of indirect and direct methods in detail and give some classical VSLAM open-source algorithms. In addition, we focus on the development of semantic VSLAM based on deep learning. Starting with typical neural networks CNN and RNN, we summarize the improvement of neural networks for the VSLAM system in detail. Later, we focus on the help of target detection and semantic segmentation for VSLAM semantic information introduction. We believe that the development of the future intelligent era cannot be without the help of semantic technology. Introducing deep learning into the VSLAM system to provide semantic information can help robots better perceive the surrounding environment and provide people with higher-level help.

show abstract

Section: Semantic With Locationmentioning

confidence: 99%

An Overview on Visual SLAM: From Tradition to Semantic

Chen

Shang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…State-of-the-art systems: The most similar state-of-the-art approaches to EnvSLAM are CNN-SLAM [42], EdgeSLAM [9], and [53]. CNN-SLAM is the first work integrating deep learning into a monocular SLAM, namely, LSD-SLAM.…”

Section: Semantic Slammentioning

confidence: 99%

“…The employed neural network is Mask R-CNN, while the selected SLAM is ORB-SLAM. In the end, [53] is the approach that is most similar to ours. Its goal is to create a semantic 3D map by combining the point cloud generated by ORB-SLAM2 with the semantic knowledge coming from PSPNet-101 [55][56][57][58].…”

Section: Semantic Slammentioning

confidence: 99%

“…We can also relate our approach to the work in [53], where GPS is used to associate landmarks from Google Maps in the created semantic map. Our goal is different, as they use the GPS to build a topological map, while we employ it to improve the segmentation result.…”

Section: Gps Integrationmentioning

confidence: 99%

“…EdgeSLAM is another similar approach, but it is noteworthy that, while it achieves real-time performance using the slow Mask R-CNN and running it on an external server, we focused on efficient neural networks and avoided to defer computation externally, as this depends on the availability of a good network connection. The most similar system to EnvSLAM is [53], as it is also based on ORB-SLAM2. We also improved upon this method by using an efficient neural network instead of the slower PSPNet-101 [55] and by performing inference only on keyframes.…”

Section: Namementioning

confidence: 99%

See 2 more Smart Citations

EnvSLAM: Combining SLAM Systems and Neural Networks to Improve the Environment Fusion in AR Applications

Marchesi

Eichhorn

Plecher

et al. 2021

IJGI

View full text Add to dashboard Cite

Augmented Reality (AR) has increasingly benefited from the use of Simultaneous Localization and Mapping (SLAM) systems. This technology has enabled developers to create AR markerless applications, but lack semantic understanding of their environment. The inclusion of this information would empower AR applications to better react to the surroundings more realistically. To gain semantic knowledge, in recent years, focus has shifted toward fusing SLAM systems with neural networks, giving birth to the field of Semantic SLAM. Building on existing research, this paper aimed to create a SLAM system that generates a 3D map using ORB-SLAM2 and enriches it with semantic knowledge originated from the Fast-SCNN network. The key novelty of our approach is a new method for improving the predictions of neural networks, employed to balance the loss of accuracy introduced by efficient real-time models. Exploiting sensor information provided by a smartphone, GPS coordinates are utilized to query the OpenStreetMap database. The returned information is used to understand which classes are currently absent in the environment, so that they can be removed from the network’s prediction with the goal of improving its accuracy. We achieved 87.40% Pixel Accuracy with Fast-SCNN on our custom version of COCO-Stuff and showed an improvement by involving GPS data for our self-made smartphone dataset resulting in 90.24% Pixel Accuracy. Having in mind the use on smartphones, the implementation aimed to find a trade-off between accuracy and efficiency, making the system achieve an unprecedented speed. To this end, the system was carefully designed and a strong focus on lightweight neural networks is also fundamental. This enabled the creation of an above real-time Semantic SLAM system that we called EnvSLAM (Environment SLAM). Our extensive evaluation reveals the efficiency of the system features and the operability in above real-time (48.1 frames per second with an input image resolution of 640 × 360 pixels). Moreover, the GPS integration indicates an effective improvement of the network’s prediction accuracy.

show abstract