Faster than FAST: GPU-Accelerated Frontend for High-Speed VIO

Abstract

The recent introduction of powerful embedded graphics processing units (GPUs) has allowed for unforeseen improvements in real-time computer vision applications. It has enabled algorithms to run onboard, well above the standard video rates, yielding not only higher information processing capability, but also reduced latency. This work focuses on the applicability of efficient low-level, GPU hardware-specific instructions to improve on existing computer vision algorithms in the field of visualinertial odometry (VIO). While most steps of a VIO pipeline work on visual features, they rely on image data for detection and tracking, of which both steps are well suited for parallelization. Especially non-maxima suppression and the subsequent feature selection are prominent contributors to the overall image processing latency. Our work first revisits the problem of nonmaxima suppression for feature detection specifically on GPUs, and proposes a solution that selects local response maxima, imposes spatial feature distribution, and extracts features simultaneously. Our second contribution introduces an enhanced FAST feature detector that applies the aforementioned nonmaxima suppression method. Finally, we compare our method to other state-of-the-art CPU and GPU implementations, where we always outperform all of them in feature tracking and detection, resulting in over 1000fps throughput on an embedded Jetson TX2 platform. Additionally, we demonstrate our work integrated into a VIO pipeline achieving a metric state estimation at ∼200fps.

Toward Efficient and Robust Multiple Camera Visual-inertial Odometry

“…Recently, several works have utilized GPUs to improve the VIO frontend video processing capability and reduce latency. The commonly used feature detectors, such as Shi-Tomasi [14], Harris [15], and FAST [16] are implemented in CUDA Visual Library (VILIB) [9] for fast feature extraction. VILIB applies efficient low-level, GPU hardware-specific programming for feature detection and feature tracking.…”

“…For each video sequence, we conduct five trials for both the original VINS-Mono and our VPI enhanced VINS-Mono with Harris corner (VINS-VPI-Harris). In addition, we deploy another GPU enhanced front-end from VILIB [9] into VINS-Mono for comparison. In VILIB, we test two feature detectors, Harris and FAST [19].…”

“…To address the above two problems when improving the efficiency and robustness of VIO systems, we mainly focus on incorporating the graphics processing units (GPUs) enhanced modules into the VIO algorithms. Recently, the powerful embedded GPUs have been utilized as the front-end of VIO to improve the information processing capability [9]. In this paper, we first incorporate the Vision Programming Interface (VPI) toolbox [10] to propose a new GPU accelerated VIO front-end, which significantly reduces the CPU usage and maintains the state estimation efficiency and accuracy.…”

Embedded Vision for Self-Driving on Forest Roads

“…where L S and L O are the semantic and instance softmax cross entropy losses, respectively, and α and β are the corresponding weight factors. The handcrafted pipeline is used for computing FAST keypoint features which are tracked in the input video streams using the pyramidal approximation of the Lucas-Kanade feature tracker [7]. In this case, no training is required, since FAST is used to compute the probable corner points.…”

Enhanced Design, Optimal Tuning, and Parallel Real-Time Implementation of Monocular Visual-Inertial-GNSS Navigation System

“…In [107], a novel local maximum search technique (known as non-maxima suppression in the computer vision community) has been introduced upon which a GPU implementation of highly parallelizable FAST features [42] detection algorithm has been proposed. The GPU-accelerated FAST feature detection was then integrated into the visual frontend of an optimization-based VIN system.…”