Multi-Task Deep Learning for Pedestrian Detection, Action Recognition and Time to Cross Prediction

Abstract

A pedestrian detection system is a crucial component of advanced driver assistance systems since it contributes to road flow safety. The safety of traffic participants could be significantly improved if these systems could also predict and recognize pedestrian's actions, or even estimate the time, for each pedestrian, to cross the street. In this paper, we focus not only on pedestrian detection and pedestrian action recognition but also on estimating if the pedestrian's action presents a risky situation according to time to cross the street. We propose 1) a pedestrian detection and action recognition component based, on RetinaNet; 2) an estimation of the time to cross the street for multiple pedestrians using a recurrent neural network. For each pedestrian, the recurrent network estimates the pedestrian's action intention in order to predict the time to cross the street. We based our experiments on the JAAD dataset, and show that integrating multiple pedestrian action tags for the detection part when merge with a recurrent neural network (LSTM) allows a significant performance improvement.INDEX TERMS Action recognition, deep learning, pedestrian detection, time-to-cross estimation.

Pedestrian Street-Cross Action Recognition in Monocular Far Infrared Sequences

“…For computing the gait information we have employed the pose extraction algorithm proposed by [76], [77], [78]. AlexNet is trained for gait (walking / Method Accuracy SVM and gait information [46] 88.75% LSTM + bounding box information [79] 80.5% ACF pedestrian detector [6] [46] and the model is modified in order to provide features on top of which a SVM classifier is trained. We have used as context the road information provided by the semantic segmentation module.…”

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

“…Similarly, Saleh et al [52] propose to predict the intended actions of pedestrians based on a spatio-temporal DenseNet model. Pop et al [8] propose to extract spatial information with convolutive layers, then consider temporal dynamics with recurrent layers and propose a new metric for pedestrians dynamics evaluation: the time to cross (TTC) prediction. Some works are based on state-of-the-art generative methods in deep-learning, focusing on the future representation of the action, and then classify it in its globality: Gujjar et al [53] and Chaabane et al [54] process the crossing actions classification by feeding the predicted frames of their future frame prediction auto-encoder network into a classification network.…”

“…Human action recognition applied to video is a difficult research topic due to the great variation and complexity of the input data. Currently, the main modalities used for these tasks include RGB videos in their entirety [1][2][3][4], optical flow [5][6][7][8] and skeleton form modeling [9][10][11][12]. The latter requires, prior to the action classification, an approach to estimate the human pose [13].…”

LiDAR-Based Multi-Task Road Perception Network for Autonomous Vehicles

“…In addition, multi-task deep learning [10], [23], [25]- [27] has attracted the attention of researchers due to its potential to boost the performance of each individual task and improve the efficiency of the total network. In RBNet [10], a Bayesian model is implemented, and the RBNet can learn to estimate the road and the road boundary simultaneously.…”