Behavioral Pedestrian Tracking Using a Camera and LiDAR Sensors on a Moving Vehicle

Abstract

In this paper, we present a novel 2D–3D pedestrian tracker designed for applications in autonomous vehicles. The system operates on a tracking by detection principle and can track multiple pedestrians in complex urban traffic situations. By using a behavioral motion model and a non-parametric distribution as state model, we are able to accurately track unpredictable pedestrian motion in the presence of heavy occlusion. Tracking is performed independently, on the image and ground plane, in global, motion compensated coordinates. We employ Camera and LiDAR data fusion to solve the association problem where the optimal solution is found by matching 2D and 3D detections to tracks using a joint log-likelihood observation model. Each 2D–3D particle filter then updates their state from associated observations and a behavioral motion model. Each particle moves independently following the pedestrian motion parameters which we learned offline from an annotated training dataset. Temporal stability of the state variables is achieved by modeling each track as a Markov Decision Process with probabilistic state transition properties. A novel track management system then handles high level actions such as track creation, deletion and interaction. Using a probabilistic track score the track manager can cull false and ambiguous detections while updating tracks with detections from actual pedestrians. Our system is implemented on a GPU and exploits the massively parallelizable nature of particle filters. Due to the Markovian nature of our track representation, the system achieves real-time performance operating with a minimal memory footprint. Exhaustive and independent evaluation of our tracker was performed by the KITTI benchmark server, where it was tested against a wide variety of unknown pedestrian tracking situations. On this realistic benchmark, we outperform all published pedestrian trackers in a multitude of tracking metrics.

Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework Using Visual and Depth Cues

“…We might note, however, that this approach disregards object's dimensions, yielding a higher average error for larger objects, providing only a rough estimate of their position. Also, for dynamic classes (e.g., humans), tracking becomes harder, requiring more robust filtering approaches, faster processing speeds, and likely taking the object's appearance into consideration [44] [45].…”

People Tracking by Cooperative Fusion of RADAR and Camera Sensors

“…Thus, during tracking the rich RADAR information is fully exploited. Whereas authors in [7] use a constant velocity model for pedestrian motion, we adopt our behavioral motion model from [10]. We solve the data association problem using the Hungarian algorithm [11] while track management is done using a Markov Decision Process approach similar to [12].…”

Efficient CORDIC Iteration Design of LiDAR Sensors’ Point-Cloud Map Reconstruction Technology

“…Because autonomous driving is the future trend and Light Detection and Ranging (LiDAR) is the core of autonomous driving, one of the key themes in autonomous car research in LiDAR [2][3][4][5]. Dimitrievski considers that LiDAR is 360 degrees surround sensor while the camera is a single view frontal view sensor [2]. Therefore, a LiDAR sensor can detect objects that are located on the sides and back of the ego vehicle [2].…”

“…Dimitrievski considers that LiDAR is 360 degrees surround sensor while the camera is a single view frontal view sensor [2]. Therefore, a LiDAR sensor can detect objects that are located on the sides and back of the ego vehicle [2]. Dimitrievski uses the results of 3D LiDAR measurements in the data association function, resulting in a notable increase in robustness to person-to-person and person-to-background occlusions [2].…”

“…Therefore, a LiDAR sensor can detect objects that are located on the sides and back of the ego vehicle [2]. Dimitrievski uses the results of 3D LiDAR measurements in the data association function, resulting in a notable increase in robustness to person-to-person and person-to-background occlusions [2]. Moreover, Zhang adopts LiDAR sensors to perform "vehicle detection using probability hypothesis density filter [3]."…”