Pose Based Action Recognition of Vulnerable Road Users Using Recurrent Neural Networks

Kress, Viktor; Schreck, Steven; Zernetsch, Stefan; Doll, Konrad; Sick, Bernhard

doi:10.1109/ssci47803.2020.9308462

Cited by 6 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the real world is 3D. Therefore, it is worth investigating the application of 3D poses [28], [29] in pedestrian crossing intention prediction using spatial-temporal graph convolutional networks.…”

Section: Discussionmentioning

confidence: 99%

“…Specifically, 2D skeleton data with 18 joints generated by OpenPose [27] is used. The Note that some studies employ 3D poses in pedestrian action recognition and prediction [28], [29]. However, in this study, to ensure a fair performance comparison with the methods in the benchmark [20], we utilize 2D poses to show the effectiveness of spatial-temporal GCNs in pedestrian crossing intention prediction.…”

Section: Skeleton Information Used In This Workmentioning

confidence: 99%

“…3. These joints include: 0-Nose, 1-Neck, 2-Right Shoulder, 3-Right Elbow, 4-Right Wrist, 5-Left Shoulder, 6-Left Elbow, 7-Left Wrist, 8-Right Hip, 9-Right Knee, 10-Right Ankle, 11-Left Hip, 12-Left Knee, 13-Left Ankle, 14-Right Eye, 15-Left Eye, 16-Right Ear, 17-Left Ear.Note that some studies employ 3D poses in pedestrian action recognition and prediction[28],[29]. However, in this study, to ensure a fair performance comparison with the methods in the benchmark[20], we utilize 2D poses to show the effectiveness of spatial-temporal GCNs in pedestrian crossing intention prediction.…”

mentioning

confidence: 99%

See 2 more Smart Citations

ST CrossingPose: A Spatial-Temporal Graph Convolutional Network for Skeleton-Based Pedestrian Crossing Intention Prediction

Zhang

Angeloudis

Demiris

2022

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

Pedestrian crossing intention prediction is crucial for the safety of pedestrians in the context of both autonomous and conventional vehicles and has attracted widespread interest recently. Various methods have been proposed to perform pedestrian crossing intention prediction, among which the skeleton-based methods have been very popular in recent years. However, most existing studies utilize manually designed features to handle skeleton data, limiting the performance of these methods. To solve this issue, we propose to predict pedestrian crossing intention based on spatial-temporal graph convolutional networks using skeleton data (ST CrossingPose). The proposed method can learn both spatial and temporal patterns from skeleton data, thus having a good feature representation ability. Extensive experiments on a public dataset demonstrate that the proposed method achieves very competitive performance in predicting crossing intention while maintaining a fast inference speed. We also analyze the effect of several factors, e.g., size of pedestrians, time to event, and occlusion, on the proposed method.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Skeleton Information Used In This Workmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

ST CrossingPose: A Spatial-Temporal Graph Convolutional Network for Skeleton-Based Pedestrian Crossing Intention Prediction

Zhang

Angeloudis

Demiris

2022

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

show abstract

“…In our future work, we plan to combine our approach with more sophisticated methods that incorporate social and physical constraints [8]. Furthermore, we will compare our method to models based on body joints, especially whether our approach can reduce the needed input length [6]. We also plan to extend our approach from a deterministic forecast to a probabilistic forecast, to evaluate whether our approach can create more reliable forecasts than an approach solely based on trajectories [22].…”

Section: Discussionmentioning

confidence: 99%

“…The joint trajectories are used in combination with balanced Gaussian process dynamical models (GPDM) to detect basic movements and forecast trajectories of pedestrians. Kress et al use joint trajectories in combination with gated recurrent units (GRU) to forecast VRU trajectories [6]. Using the additional information from body joints, they are able to forecast more accurate positions with shorter observation periods than a solely trajectory based method.…”

Section: B Related Workmentioning

confidence: 99%

Cyclist Trajectory Forecasts by Incorporation of Multi-View Video Information

Zernetsch¹,

Trupp²,

Kress³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This article presents a novel approach to incorporate visual cues from video-data from a wide-angle stereo camera system mounted at an urban intersection into the forecast of cyclist trajectories. We extract features from image and optical flow (OF) sequences using 3D convolutional neural networks (3D-ConvNet) and combine them with features extracted from the cyclist's past trajectory to forecast future cyclist positions. By the use of additional information, we are able to improve positional accuracy by about 7.5 % for our test dataset and by up to 22 % for specific motion types compared to a method solely based on past trajectories. Furthermore, we compare the use of image sequences to the use of OF sequences as additional information, showing that OF alone leads to significant improvements in positional accuracy. By training and testing our methods using a real-world dataset recorded at a heavily frequented public intersection and evaluating the methods' runtimes, we demonstrate the applicability in real traffic scenarios. Our code and parts of our dataset are made publicly available.

show abstract

Detecting Intentions of Vulnerable Road Users Based on Collective Intelligence as a Basis for Automated Driving

Zernetsch,

Kress,

Bieshaar

et al. 2024

Cooperatively Interacting Vehicles

View full text Add to dashboard Cite

The project Detecting Intentions of Vulnerable Road Users Based on Collective Intelligence as a Basis for Automated Driving (DeCoInt$$^2$$ 2 ) focuses on detecting the intentions of vulnerable road users (VRUs) in automated driving using cooperative technologies. Especially in urban areas, VRUs, e.g., pedestrians and cyclists, will continue to play an essential role in mixed traffic. For an accident-free and highly efficient traffic flow with automated vehicles, it is vital to perceive VRUs and their intentions and analyze them similarly to humans when driving and forecasting VRU trajectories. Doing this reliably and robustly with a multimodal sensor system (e.g., cameras, LiDARs, accelerometers, and gyroscopes in mobile devices) in real-time is a big challenge. We follow a holistic, cooperative approach to recognize humans’ movements and forecast their trajectories. Heterogeneous open sets of agents, i.e., collaboratively interacting vehicles, infrastructure, and VRUs equipped with mobile devices, exchange information to determine individual models of their surrounding environment, allowing an accurate and reliable forecast of VRU basic movements and trajectories. The collective intelligence of cooperating agents resolves occlusions, implausibilities, and inconsistencies. We developed new methods by considering and combining novel signal processing and modeling techniques with machine learning-based pattern recognition approaches. The cooperation between agents happens on several levels: the VRU perception level, the level of recognized trajectories, or the level of already detected intentions.

show abstract

Pose Based Action Recognition of Vulnerable Road Users Using Recurrent Neural Networks

Cited by 6 publications

References 17 publications

ST CrossingPose: A Spatial-Temporal Graph Convolutional Network for Skeleton-Based Pedestrian Crossing Intention Prediction

ST CrossingPose: A Spatial-Temporal Graph Convolutional Network for Skeleton-Based Pedestrian Crossing Intention Prediction

Cyclist Trajectory Forecasts by Incorporation of Multi-View Video Information

Detecting Intentions of Vulnerable Road Users Based on Collective Intelligence as a Basis for Automated Driving

Contact Info

Product

Resources

About