Orientation- and Scale-Invariant Multi-Vehicle Detection and Tracking from Unmanned Aerial Videos

Abstract

Along with the advancement of light-weight sensing and processing technologies, unmanned aerial vehicles (UAVs) have recently become popular platforms for intelligent traffic monitoring and control. UAV-mounted cameras can capture traffic-flow videos from various perspectives providing a comprehensive insight into road conditions. To analyze the traffic flow from remotely captured videos, a reliable and accurate vehicle detection-and-tracking approach is required. In this paper, we propose a deep-learning framework for vehicle detection and tracking from UAV videos for monitoring traffic flow in complex road structures. This approach is designed to be invariant to significant orientation and scale variations in the videos. The detection procedure is performed by fine-tuning a state-of-the-art object detector, You Only Look Once (YOLOv3), using several custom-labeled traffic datasets. Vehicle tracking is conducted following a tracking-by-detection paradigm, where deep appearance features are used for vehicle re-identification, and Kalman filtering is used for motion estimation. The proposed methodology is tested on a variety of real videos collected by UAVs under various conditions, e.g., in late afternoons with long vehicle shadows, in dawn with vehicles lights being on, over roundabouts and interchange roads where vehicle directions change considerably, and from various viewpoints where vehicles’ appearance undergo substantial perspective distortions. The proposed tracking-by-detection approach performs efficiently at 11 frames per second on color videos of 2720p resolution. Experiments demonstrated that high detection accuracy could be achieved with an average F1-score of 92.1%. Besides, the tracking technique performs accurately, with an average multiple-object tracking accuracy (MOTA) of 81.3%. The proposed approach also addressed the shortcomings of the state-of-the-art in multi-object tracking regarding frequent identity switching, resulting in a total of only one identity switch over every 305 tracked vehicles.

A Systematic Review of Drone Based Road Traffic Monitoring System

“…In addition, [68] carried out traffic monitoring task in an urban area, and used Mask R-CNN framework [102] to detect the vehicles from videos recorded from a fixed positioned UAV. Different complex road scenes such as long vehicles shadows, vehicles with lights-on, roundabouts, interchange roads and different viewpoints were considered in [94] where YOLOv3 framework [25] was employed for the vehicle detection task. The performance of system was invariant to the orientation and scale variations in UAV videos.…”

“…However, in [12] the main focus was given on multiple vehicles speed estimation by taking into account tracking and motion estimation. From a different perspective, in [94] vehicle tracking is achieved by performing vehicle Re-IDentification (Re-ID) with deep features and motion estimation with KF technique. As stated in [109], DCF tracker alone is not only unable to handle the occlusion problem, but it also can not provide robust tracking.…”

Deep Learning for UAV-based Object Detection and Tracking: A Survey

“…Jin et al [177] proposed online MOT with Siamese network and optical flow (Siamese-OF). Shuai et al [178] proposed MOT [150] Orientation, scale UAVDT Remote Sens. 2019 -Flow-tracker [151] ID Switches, error detection VisDrone-MOT ICCV 2019 -TNT [152] Camera motion, occlusion, pose variation VisDrone-MOT, Own ACM-MM 2019 -HMTT [153] Target motion, shape, appearance changes VisDrone-MOT ICCV 2019 -Yang et al [154] Target position changes Own RS 2019 https://frank804.github.io/ GGD [155] False alarms, missed detections VisDrone-MOT ICCV 2019 https://github.com/hakanardo/ggdtrack COMET [156] Small object UAVDT, VisDrone-MOT, Small-90 ICCV 2019 -Self-balance [157] Appearance, motion UAVDT Multimedia Asia 2019 -Abughalieh et al [117] Low detailed targets DARPA, VIVID, Own Multimed.…”

“…OSIM [150] YOLOv3+Kalman filtering 2720 × 1530 Workstation(NVIDIA GeForce GTX 1080 Ti/-) 30 2019 +deep appearance feature Workstation(Intel UHD Graphics 630/-) Self-balance [157] LSTM 1080 × 540 Workstation(NVIDIA Titan X/32GB) -2019 Flow-tracker [151] Optical Flownet+IOU -Workstation(NVIDIA GTX 1080Ti/-) 5 2019 HMTT [153] CenterNet+IOU+OSNet ---2019 Yang et al [154] YOLOv3 + dense-trajectory-Voting 1920 × 1080 Workstation(NVIDIA GeForce GTX1080Ti/6GB) 8.6 2019 GGD [155] Faster RCNN+GGD -Workstation(NVIDIA GTX 1080/-) -2019 COMET [156] ResNet-50+Two-stream network 1080 × 540 Workstation(NVIDIA Tesla V100/16GB) 24 2019 Abughalieh et al [117] FAST 320 × 240 Laptop(Core i7-2670QM/6GB) 26.3 2019 Abughalieh et al [117] FAST 320 × 240 Embedded(Raspberry Pi 2/1GB) 10.8 2019 IPGAT [159] SiameseNet+LSTM+CGAN -Workstation(NVIDIA Titan X/32GB) -2020 Kapania et al [160] YOLOv3+RetinaNet ---2020 PAS tracker [161] Cascade R-CNN+Similarity ---2020 DAN [77] RetinaNet+DeepSORT 1500 × 1000 -2020 DQN [162] Faster R-CNN ---2021 Youssef et al [80] Cascade visible-thermal infrared cameras equipped with drones. In label terms, the bounding box is not limited to horizontal bounding box strongly dependent on robustness to direction, even have oriented bounding box, e.g., in the Vehicle Dataset.…”

Learning Response-Consistent and Background-Suppressed Correlation Filters for Real-Time UAV Tracking

“…Target tracking is a research hotspot in machine vision and has been widely used in video surveillance, security, inspection, human–computer interaction, etc. In recent years, with the rapid development of unmanned aerial vehicles (UAV), target tracking has become one of its basic functions, which has been applied in aerial photogrammetry [ 1 ], autonomous landing [ 2 ], target location [ 3 ], aerial surveillance [ 4 ], object detection [ 5 ], etc. However, target tracking in UAVs has still encountered many challenges, such as background clutter, similar object, partial/full occlusion, and fast motion.…”