Real-time 3D Deep Multi-Camera Tracking

Abstract

Tracking a crowd in 3D using multiple RGB cameras is a challenging task. Most previous multi-camera tracking algorithms are designed for offline setting and have high computational complexity. Robust real-time multi-camera 3D tracking is still an unsolved problem. In this work, we propose a novel end-to-end tracking pipeline, Deep Multi-Camera Tracking (DMCT), which achieves reliable real-time multi-camera people tracking. Our DMCT consists of 1) a fast and novel perspective-aware Deep GroudPoint Network, 2) a fusion procedure for groundplane occupancy heatmap estimation, 3) a novel Deep Glimpse Network for person detection and 4) a fast and accurate online tracker. Our design fully unleashes the power of deep neural network to estimate the "ground point" of each person in each color image, which can be optimized to run efficiently and robustly. Our fusion procedure, glimpse network and tracker merge the results from different views, find people candidates using multiple video frames and then track people on the fused heatmap. Our system achieves the state-of-the-art tracking results while maintaining real-time performance. Apart from evaluation on the challenging WILDTRACK dataset, we also collect two more tracking datasets with high-quality labels from two different environments and camera settings. Our experimental results confirm that our proposed real-time pipeline gives superior results to previous approaches.

Generalizable Multi-Camera 3D Pedestrian Detection

“…Alternatively, MVDet [10] aggregates the multi-view people detection information by applying a feature perspective transform to place all ground heatmaps (and later locations) of pedestrians in the same coordinate space. Similarly, DMCT [21] proposes a perspective-aware network, which produces distorted detection blobs (related to the camera's perspective). This is followed by a fusion procedure for ground-plane occupancy heatmap estimation and the use of a Deep Glimpse Network for person detection.…”

“…• AH, which computes an average heatmap as described by You and Jiang [21]. It obtains each camera's heatmaps by considering a non-normalized Gaussian kernel in world ground plane coordinates centered at each ground point with a radius of 0.8m and σ = 10.1.…”

“…Similar to many other existing multi-camera 3D pedestrian detection methods in the literature [10,21,23,16,1], our proposed approach is restricted to the ground plane. Therefore, it cannot correctly estimate the 3D location of people who are not standing on the ground (e.g., jumping).…”

LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Tracking

“…GMLP [31] is inspired by the probabilistic approach [13] and jointly uses CNNs and Conditional Random Fields to model explicitly an occupancy volume map given detections estimated from multiple cameras. More recently [46] (DMCT) propose deep learning to directly compute the occupancy volume by fusing feature maps extracted from CNNs at multi-camera views.…”

“…As a result, these errors will propagate throughout the tracking graph, affecting total performance. The centralized representation approach [46,48] on the other hand, is not plagued by such obstacles since each node in the tracking graph is an occupancy map (not a tracklet), which is estimated from all detections at each timeframe. Unfortunately, the cost of the data association step is increased due to a huge state space of variables and integrating advances from single-camera methods is more complicated.…”