Joint detection and tracking in videos with identification features

Abstract

Recent works have shown that combining object detection and tracking tasks, in the case of video data, results in higher performance for both tasks, but they require a high frame-rate as a strict requirement for performance. This assumption is often violated in real-world applications, when models run on embedded devices, often at only a few frames per second. Videos at low frame-rate suffer from large object displacements. Here re-identification features may support to match large-displaced object detections, but current joint detection and re-identification formulations degrade the detector performance, as these two are contrasting tasks. In the real-world application having separate detector and re-id models is often not feasible, as both the memory and runtime effectively double. Towards robust long-term tracking applicable to reduced-computational-power devices, we propose the first joint optimization of detection, tracking and re-identification features for videos. Notably, our joint optimization maintains the detector performance, a typical multi-task challenge. At inference time, we leverage detections for tracking (tracking-by-detection) when the objects are visible, detectable and slowly moving in the image. We leverage instead re-identification features to match objects which disappeared (e.g. due to occlusion) for several frames or were not tracked due to fast motion (or low-frame-rate videos). Our proposed method reaches the state-of-the-art on MOT, it ranks 1st in the UA-DETRAC'18 tracking challenge among online trackers, and 3rd overall.

ECCNet: Efficient chained centre network for real‐time multi‐category vehicle tracking and vehicle speed estimation

“…An end‐to‐end joint neural network [46] addresses detection, tracking and re‐identification altogether, which detects objects, provides tracking associations across frames and estimates id‐features, to match objects across frames further apart. For a fairer comparison, Model2 [46], this version without identification features in the IoU estimation, is chosen as the comparison model.…”

OPTIMIZING PEDESTRIAN TRACKING FOR ROBUST PERCEPTION WITH YOLOv8 AND DEEPSORT

“…The tracking of multiple objects (MOT) is an active field of research in computer vision. Recently, two main approaches have been the focus of research: tracking by detection (Bergmann et al, 2019;Pang et al, 2020;Peng et al, 2020;Wang et al, 2020;) and joint detection and tracking (Munjal et al, 2020;Feng et al, 2023;Wang et al, 2021). Joint detection and tracking methods detect and track objects within a single model, utilizing visual appearance to locate objects within images.…”

Joint 3D Object Detection and Tracking Using Spatio-Temporal Representation of Camera Image and LiDAR Point Clouds

“…Various JoDT methods have been proposed in (Zhang et al 2021;Wang, Kitani, and Weng 2020;Peng et al 2020;Hu et al 2019;Shenoi et al 2020;Kim and Kim 2016;Kieritz, Hubner, and Arens 2018;Ke et al 2019;Munjal et al 2020). In the early phase, the idea of JoDT was adopted for 2D MOT.…”

“…The appearance cues and motion context identified by the detector were used to perform tracking. The 2D MOT methods in this category include MPNTrack (Brasó and Leal-Taixé 2020), RNN tracker (Kieritz, Hubner, and Arens 2018), CDT (Kim and Kim 2016), PredNet (Munjal et al 2020), and Chained-Tracker (Peng et al 2020). The JoDT methods for 3D MOT include the mono3DT (Hu et al 2019) and JRMOT (Shenoi et al 2020).…”