Navigating the Landscape for Real-Time Localization and Mapping for Robotics and Virtual and Augmented Reality

Abstract

Visual understanding of 3D environments in realtime, at low power, is a huge computational challenge. Often referred to as SLAM (Simultaneous Localisation and Mapping), it is central to applications spanning domestic and industrial robotics, autonomous vehicles, virtual and augmented reality. This paper describes the results of a major research effort to assemble the algorithms, architectures, tools, and systems software needed to enable delivery of SLAM, by supporting applications specialists in selecting and configuring the appropriate algorithm and the appropriate hardware, and compilation pathway, to meet their performance, accuracy, and energy consumption goals. The major contributions we present are (1) tools and methodology for systematic quantitative evaluation of SLAM algorithms, (2) automated, machine-learning-guided exploration of the algorithmic and implementation design space with respect to multiple objectives, (3) end-to-end simulation tools to enable optimisation of heterogeneous, accelerated architectures for the specific algorithmic requirements of the various SLAM algorithmic approaches, and (4) tools for delivering, where appropriate, accelerated, adaptive SLAM solutions in a managed, JIT-compiled, adaptive runtime context. Performance EvaluationRuntime Architecture Compiler and Algorithm Design Space Exploration -Machine Learning Fig. 1: The objective of the paper is to create a pipeline that aligns computer vision requirements with hardware capabilities. The paper's focus is on three layers: algorithms, compiler and runtime, and architecture. The goal is to develop a system that allows us to achieve power and energy efficiency, speed and runtime improvement, and accuracy/robustness at each layer and also holistically through design space exploration and machine learning techniques.

Characterizing Visual Localization and Mapping Datasets

“…Similar benchmarking works have been performed in [5] and [6]. As the application of SLAM algorithms in robotics and computer vision is growing, it is becoming apparent that a more sophisticated approach to benchmarking is needed [7]. In this paper we develop ideas from statistics to propose Department of Computing, Imperial College London, UK § Department of Computer Science, University of Bath, UK novel metrics to label datasets in terms of the motion, structure, and appearance qualities which are important to SLAM performance.…”

SLAMBench 3.0: Systematic Automated Reproducible Evaluation of SLAM Systems for Robot Vision Challenges and Scene Understanding

“…Similar to SemanticFusion, ORB-SLAM2-CNN [53] is based on ORB-SLAM2 [42], projecting the segmentation of a modified version of MobileNet [29] to label the keypoints of ORB-SLAM2-generated map. Thus, the key difference from SemanticFusion is that this algorithm produces a labelled sparse map, rather than a dense one.…”

“…• Enhancements that allow joint evaluation of quality for both reconstruction and semantic segmentation, using the NYU RGB-Dv2 [44] and ScanNet [8] datasets. This is demonstrated on two systems: SemanticFusion [40] and ORB-SLAM2-CNN [53], which construct labelled dense and sparse scene maps, respectively.…”

Towards Identification of Packaged Products via Computer Vision

“…one second by allowing the neural net to detect a product on as many images as it can process in this time. We can use spatial information from algorithms such as SLAM [2,29,37], video object tracking [20,24] or optical flow to track [15,28] the position over multiple frames. The prediction scores from those multiple frames are then average-pooled to choose the maximumconfidence prediction across all frames (within one second).…”