CAPformer: Pedestrian Crossing Action Prediction Using Transformer

Abstract

Anticipating pedestrian crossing behavior in urban scenarios is a challenging task for autonomous vehicles. Early this year, a benchmark comprising JAAD and PIE datasets have been released. In the benchmark, several state-of-the-art methods have been ranked. However, most of the ranked temporal models rely on recurrent architectures. In our case, we propose, as far as we are concerned, the first self-attention alternative, based on transformer architecture, which has had enormous success in natural language processing (NLP) and recently in computer vision. Our architecture is composed of various branches which fuse video and kinematic data. The video branch is based on two possible architectures: RubiksNet and TimeSformer. The kinematic branch is based on different configurations of transformer encoder. Several experiments have been performed mainly focusing on pre-processing input data, highlighting problems with two kinematic data sources: pose keypoints and ego-vehicle speed. Our proposed model results are comparable to PCPA, the best performing model in the benchmark reaching an F1 Score of nearly 0.78 against 0.77. Furthermore, by using only bounding box coordinates and image data, our model surpasses PCPA by a larger margin (F1=0.75 vs. F1=0.72). Our model has proven to be a valid alternative to recurrent architectures, providing advantages such as parallelization and whole sequence processing, learning relationships between samples not possible with recurrent architectures.

Early Pedestrian Intent Prediction via Features Estimation

“…Our proposed model suffers from a hard reduction of training samples, compared to [15] which limits its performance on larger datasets, e.g., JAAD all . Nevertheless, our model is on par with CAPformer [17] and outperforms TrouSPI-Net [19] for both subsets.…”

“…Furthermore, we compare our model, at different anticipation times, against PCPA [15] which uses fixed and overlapped observations of 0.5 s in the range [2 − 1] s earlier the event. For our purposes, this model is modified to output earlier predictions in the range [4 − 1] s. We also consider CAPformer [17] and TrouSPI-Net [19] models.…”

“…More recent works extend this anticipation time to different values with different observation lengths, in addition to developing more advanced models with multiple input features [2,3,4]. In [15], an evaluation benchmark is proposed to tackle this problem, also adopted in [16,17,18,19]. This benchmark focuses on predicting future intentions between 1.0 to 2.0 s earlier the event and uses overlapping windows of 0.5 s as motion history.…”

Testing predictive automated driving systems: lessons learned and future recommendations

“…For pedestrians, body and facial keypoints detectors [35] act as the core for prediction systems. Deep learning approaches use body keypoints to anticipate changes in pedestrian motion patterns [36] and also to predict the intention of crossing from the sidewalk or at a crosswalk [37], [38]. Face key points are of paramount importance for the detection of crossing intention, being eye contact a powerful non-verbal channel of communication often used to express intention to drivers.…”

Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction

“…In this field, 2D poses [2], [3], pedestrian bounding boxes [4], optical flow [5], scene context [6], vehicles speeds [7], trajectories [8], ego-motion of vehicles [7] are utilized in previous works. In the meantime, the deep learning models, such as I3D [5], LSTM/RNN-based temporal models [8], [9], as well as the transformers [10] are adopted in recent years. However, because of the high-mobility of pedestrian, the prediction results of previous works do not approve each other [11], especially for the starting time when the pedestrians show a small scale.…”