Semantics-STGCNN: A Semantics-guided Spatial-Temporal Graph Convolutional Network for Multi-class Trajectory Prediction

Abstract

Predicting the movement trajectories of multiple classes of road users in real-world scenarios is a challenging task due to the diverse trajectory patterns. While recent works of pedestrian trajectory prediction successfully modelled the influence of surrounding neighbours based on the relative distances, they are ineffective on multi-class trajectory prediction. This is because they ignore the impact of the implicit correlations between different types of road users on the trajectory to be predicted-for example, a nearby pedestrian has a different level of influence from a nearby car. In this paper, we propose to introduce class information into a graph convolutional neural network to better predict the trajectory of an individual. We embed the class labels of the surrounding objects into the label adjacency matrix (LAM), which is combined with the velocity-based adjacency matrix (VAM) comprised of the objects' velocity, thereby generating a semantics-guided graph adjacency (SAM). SAM effectively models semantic information with trainable parameters to automatically learn the embedded label features that will contribute to the fixed velocity-based trajectory. Such information of spatial and temporal dependencies is passed to a graph convolutional and temporal convolutional network to estimate the predicted trajectory distributions. We further propose new metrics, known as Average 2 Displacement Error (aADE) and Average Final Displacement Error (aFDE), that assess network accuracy more accurately. We call our framework Semantics-STGCNN. It consistently shows superior performance to the state-of-the-arts in existing and the newly proposed metrics.

Multiclass-SGCN: Sparse Graph-Based Trajectory Prediction with Agent Class Embedding

“…We existing works [9], [16] that apply 8 observed frames (3.2 seconds) to predict the next 12 frames (4.8 seconds), then 20 samples are derived from the learnt multivariate distribution. The model was evaluated in terms of the Minimum Average Displacement Error (mADE) and the Minimum Final Displacement Error (mFDE) as in [4], as well as in terms of the Average ADE (aADE) and the Average FDE (aFDE) proposed by [10] who argued that aADE and aFDE evaluate the models more holistically. The Adam [17] optimiser was used for training, with a 0.0001 learning rate and a batch size of 256.…”

“…The Adam [17] optimiser was used for training, with a 0.0001 learning rate and a batch size of 256. To compare with Semantics-STGCNN [10], we also normalised and denormalised the input trajectory data with a scaling factor of 10. Training typically converged in around 35-45 epochs.…”

“…The proposed method was compared to 8 models in total, including the baseline Linear model, energy function based behavioral model (SF [18] ), Social-LSTM [3], Social-GAN [16], CAR-Net [19], DESIRE [20], Social-STGCNN [4] and Semantics-STGCNN [10], the existing state-of-theart model for multi-class trajectory prediction. Notably, the results of Semantics-STGCNN were evaluated using the published source code, whereas other results were provided by [10]. Results are presented in Table 1 in terms of mADE and mFDE.…”

“…We observe that the two important factors that affect the movement of an agent are the classes and velocity of neighbours. Class labels, L i t , can indicate how different classes of agents, such as pedestrian, car, cyclist, have different influences [10]. Velocity, V i t , enhances the ability of a model to capture the geometric features of agents [4].…”

“…Intuitively speaking, even if two agents have a similar velocity, human instincts would force us to pay more attention to the movements of the larger agents, such as considering car over bicycle. To address this issue, Semantics-STGCNN [10,11] considered class labels for multi-class trajectory prediction by embedding agent-label features into the velocity representations [12], ensuring that the upcoming GCN [13] aggregates both features. Nevertheless, Semantics-STGCNN still suffers from the superfluous interactions problem as it uses a densely connected graph.…”