Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

Abstract

Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on top of segmented object instances within and across video frames. Pedestrian intent, defined as the future action of crossing or not-crossing the street, is very crucial piece of information for autonomous vehicles to navigate safely and more smoothly. We approach the problem of intent prediction from two different perspectives and anticipate the intention-to-cross within both pedestrian-centric and location-centric scenarios. In addition, we introduce a new dataset designed specifically for autonomousdriving scenarios in areas with dense pedestrian populations: the Stanford-TRI Intent Prediction (STIP) dataset. Our experiments on STIP and another benchmark dataset show that our graph modeling framework is able to predict the intention-to-cross of the pedestrians with an accuracy of 79.10% on STIP and 79.28% on Joint Attention for Autonomous Driving (JAAD) dataset up to one second earlier than when the actual crossing happens. These results outperform baseline and previous work. Please refer to http://stip.stanford.edu/ for the dataset and code.Index Terms-spatiotemporal graphs, forecasting, graph neural networks, autonomous-driving. Recent work [19]-[23] introduced pedestrian intent prediction and have typically tackled the problem by observing pedestrian-specific features such as location, velocity, and

CAPformer: Pedestrian Crossing Action Prediction Using Transformer

“…In the future, some possible lines of work are detailed below: Evaluate our model in additional datasets, such as STIP [ 25 ] or TITAN [ 24 ]; Train and test models in a combination of different available behavior datasets and analyze it in an unrelated scenario to see its generalization capabilities; Research deeper in the usage of data augmentation techniques in the training of multi-branch models; Consider the development of virtual scenarios to include more crossing cases and fight data imbalance; Explore new features from datasets, such as labeled information from the environment (e.g., the relative relationship between vehicles, crosswalk position, traffic signals). These new features will be 2D or 3D, depending on the availability of datasets in the literature; Experimentation with different data cleaning strategies in training time, applying a maximum occlusion level, pedestrian minimum size, etc.…”

“…Another dataset with crossing behavior, binary annotated in this case, is STIP dataset [ 25 ]. This dataset is recorded with a multi-camera setup and includes hand-labeled bounding boxes at a low annotation rate.…”

“…A tracking algorithm is used for sequence creation and to interpolate the annotations at a higher rate. Reference [ 25 ] proposes a graph-based approach where a scene graph is created for each pedestrian sequence to benefit from environment information. After graph creation, a GRU recurrent model is used to anticipate crossing action.…”

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

“…However, the generated context features are not always applicable to current data sets and require different kinds of modalities or hardware constraints [60,61]. In that regard, Liu et al [62] propose a new data set for pedestrian intention prediction tailored to intent prediction in dense driving scenes. It defines a model based on graph convolutions to represent the spatiotemporal context of the scene where each identified object is presented as a node of a spatiotemporal graph for two different perspectives: pedestrian-centric and location-centric settings graphs.…”

STI-GAN: Multimodal Pedestrian Trajectory Prediction Using Spatiotemporal Interactions and a Generative Adversarial Network

“…Haddad et al [5] used spatiotemporal graphs to capture both the temporal and spatial correlations of pedestrian predictions and considered physical cues in a scene and the interactions between pedestrians, thereby improving the performance of trajectory prediction. In addition, Liang et al [6] and Liu et al [7] considered pedestrian-scene and pedestrian-object relationships simultaneously and incorporated pedestrian intentions to model future paths and predict human activities and locations. However, their work ignored the multimodal nature of the prediction of future pedestrian trajectories.…”