Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet

Saleh, Khaled J.; Hossny, Mohammed; Nahavandi, Saeid

doi:10.1109/icra.2019.8793991

Cited by 57 publications

(50 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Method Accuracy Alexnet + Context [15] 63.0% Alexnet + SVM [50] 74.4% Alphapose + LSTM [56] 78.0% Res-EnDec [53] 81.0% ST-DenseNet [52] 84.76% auto-encoder + Prediction [54] 86.7% Openpose + Keypoints [55] 88.0% Alexnet + SVM + Context [50] 89.4% CPN + GCN [58] 91 Overall, although SPI-Net is not that complex in its architecture, Table 3 shows that it outperforms by more than 2.5% the current state-of-the-art approach [58] based on CPN [14] for pedestrian discrete intention prediction task on the JAAD data. The confusion matrices in Table 4 also shows that SPI-Net accuracy is similar on both action classes, which demonstrates its ability to adapt to intra-class variation for skeleton-based dynamics.…”

Section: Results On Jaad Data Setmentioning

confidence: 99%

“…Afterward, they extend their model in order to take as input a sequence of consecutive cropped images of the pedestrians before they cross in order to consider the temporal coherence in short-term motions (≈0.5 s). Similarly, Saleh et al [52] propose to predict the intended actions of pedestrians based on a spatio-temporal DenseNet model. Pop et al [8] propose to extract spatial information with convolutive layers, then consider temporal dynamics with recurrent layers and propose a new metric for pedestrians dynamics evaluation: the time to cross (TTC) prediction.…”

Section: Pedestrian Intention Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

et al. 2020

View full text Add to dashboard Cite

Understanding the behaviors and intentions of humans is still one of the main challenges for vehicle autonomy. More specifically, inferring the intentions and actions of vulnerable actors, namely pedestrians, in complex situations such as urban traffic scenes remains a difficult task and a blocking point towards more automated vehicles. Answering the question “Is the pedestrian going to cross?” is a good starting point in order to advance in the quest to the fifth level of autonomous driving. In this paper, we address the problem of real-time discrete intention prediction of pedestrians in urban traffic environments by linking the dynamics of a pedestrian’s skeleton to an intention. Hence, we propose SPI-Net (Skeleton-based Pedestrian Intention network): a representation-focused multi-branch network combining features from 2D pedestrian body poses for the prediction of pedestrians’ discrete intentions. Experimental results show that SPI-Net achieved 94.4% accuracy in pedestrian crossing prediction on the JAAD data set while being efficient for real-time scenarios since SPI-Net can reach around one inference every 0.25 ms on one GPU (i.e., RTX 2080ti), or every 0.67 ms on one CPU (i.e., Intel Core i7 8700K).

show abstract

Section: Results On Jaad Data Setmentioning

confidence: 99%

Section: Pedestrian Intention Predictionmentioning

confidence: 99%

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In [ 19 ], a 3D CNN is used as a classifier at the end of a pipeline for pedestrian crossing behavior, which includes detection and tracking. The 3D convolutional model is trained with the cropped pedestrians’ bounding boxes detected.…”

Section: Related Workmentioning

confidence: 99%

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

Lorenzo

Alonso

Izquierdo

et al. 2021

Sensors

View full text Add to dashboard Cite

Anticipating pedestrian crossing behavior in urban scenarios is a challenging task for autonomous vehicles. Early this year, a benchmark comprising JAAD and PIE datasets have been released. In the benchmark, several state-of-the-art methods have been ranked. However, most of the ranked temporal models rely on recurrent architectures. In our case, we propose, as far as we are concerned, the first self-attention alternative, based on transformer architecture, which has had enormous success in natural language processing (NLP) and recently in computer vision. Our architecture is composed of various branches which fuse video and kinematic data. The video branch is based on two possible architectures: RubiksNet and TimeSformer. The kinematic branch is based on different configurations of transformer encoder. Several experiments have been performed mainly focusing on pre-processing input data, highlighting problems with two kinematic data sources: pose keypoints and ego-vehicle speed. Our proposed model results are comparable to PCPA, the best performing model in the benchmark reaching an F1 Score of nearly 0.78 against 0.77. Furthermore, by using only bounding box coordinates and image data, our model surpasses PCPA by a larger margin (F1=0.75 vs. F1=0.72). Our model has proven to be a valid alternative to recurrent architectures, providing advantages such as parallelization and whole sequence processing, learning relationships between samples not possible with recurrent architectures.

show abstract

“…Also, DenseNets are simpler and more efficient as compared to Inception networks [9]. K. Saleh [13] employed Spatiotemporal DenseNet architecture to predict pedestrians' intended action and achieved an average precision score of 0.8476. The DenseNet architecture [12] where each layer is connected to all the others within a dense block.…”

Section: ) Densenetmentioning

confidence: 99%

An Empirical Analysis of Deep Learning Architectures for Vehicle Make and Model Recognition

et al. 2021

View full text Add to dashboard Cite

Vehicle make and model recognition is an important component of the Intelligent Transport System (ITS), which plays an essential role in vehicle surveillance and traffic monitoring. In this paper, we evaluated the performance of the recent deep neural networks for vehicle recognition to identify 196 different types of vehicles based on their make, model, and year using Stanford Cars data. Transfer learning has been employed to reduce the training time and also added a dropout of 0.5 at the last dense layer before the output layer for improved generalization performance on an applied dataset. Different augmentation methods such as random erasing, horizontal flip, random crop, resize, rotate, shear, and sharpen were applied to the training dataset to increase the number of images. Besides, optimization methods namely mixup and cyclical learning rate were also investigated to train these deep neural networks to avoid overfitting. Finally, the proposed ensemble learning framework using k-fold cross-validation has resulted in the improved accuracy of image classification for the vehicle make and model recognition. The final achieved accuracy is 93.96% through ensembling homogeneous models of DenseNet201, which is higher than previous stateof-the-art approaches for the vehicle make and model recognition. Simultaneously, it has also improved the performance of other models where MobileNetV2 has shown the highest improvement with an increase of 2.8% in the overall accuracy using the proposed ensemble learning technique.INDEX TERMS deep neural networks, vehicle make and model recognition, ensemble learning, and k-fold cross-validation

show abstract

Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet

Cited by 57 publications

References 29 publications

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

An Empirical Analysis of Deep Learning Architectures for Vehicle Make and Model Recognition

Contact Info

Product

Resources

About