Deep Learning-Based Real-Time Multiple-Object Detection and Tracking from Aerial Imagery via a Flying Robot with GPU-Based Embedded Devices

Abstract

In recent years, demand has been increasing for target detection and tracking from aerial imagery via drones using onboard powered sensors and devices. We propose a very effective method for this application based on a deep learning framework. A state-of-the-art embedded hardware system empowers small flying robots to carry out the real-time onboard computation necessary for object tracking. Two types of embedded modules were developed: one was designed using a Jetson TX or AGX Xavier, and the other was based on an Intel Neural Compute Stick. These are suitable for real-time onboard computing power on small flying drones with limited space. A comparative analysis of current state-of-the-art deep learning-based multi-object detection algorithms was carried out utilizing the designated GPU-based embedded computing modules to obtain detailed metric data about frame rates, as well as the computation power. We also introduce an effective target tracking approach for moving objects. The algorithm for tracking moving objects is based on the extension of simple online and real-time tracking. It was developed by integrating a deep learning-based association metric approach with simple online and real-time tracking (Deep SORT), which uses a hypothesis tracking methodology with Kalman filtering and a deep learning-based association metric. In addition, a guidance system that tracks the target position using a GPU-based algorithm is introduced. Finally, we demonstrate the effectiveness of the proposed algorithms by real-time experiments with a small multi-rotor drone.

AB9: A neural processor for inference acceleration

“…Table 5 shows the performance when running VGG-19 [6], while Table 6 shows the performance for YOLOv2 [5] with the standard 416 × 416 image size. NVIDIA Pascal Titan X 67 GPU [5] NVIDIA Jetson TX2 7 GPU [5] NVIDIA Xavier AGX 30 GPU [12] NVIDIA GTX1080 28 GPU [12] Xilinx FPGA with DPU 25 FPGA [12] From the results, we observe that the AB9 SoC exhibits a higher fps than the other surveyed hardware. The advantage in performance can be attributed to the hardware optimizations customized for NN computations.…”

3d Photogrammetric Inspection of Risers Using Rpas and Deep Learning in Oil and Gas Offshore Platforms

“…This approach outperforms SSD and RetinaNet in terms of speed and it is competitive in terms of accuracy in detecting large objects 2 (Liu et al, 2018) (Li et al, 2020). Moreover, recent works such as (Hossain and Lee, 2019), (Sadykova et al, 2019), (Opromolla et al, 2019) and (Chen and Miao, 2020) achieved good performance in applications involving RPAs and real-time object detection using YOLOv2.…”

Enhancement Performance of Multiple Objects Detection and Tracking for Real-time and Online Applications

“…It shows good accuracy and a high apparent positive rate and a minor false positive. Blanco-Filgueira, B et al [20] implemented a visual tracing of multiple objects based on deep real-time learning using the NVIDIA Jetson TX2 device. The results highlighted the effectiveness of the algorithm under real challenging scenarios in different environmental conditions, such as low light and high contrast in the tracking phase and not consider in the detection phase.…”