SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction

Abstract

We propose advances that address two key challenges in future trajectory prediction: (i) multimodality in both training data and predictions and (ii) constant time inference regardless of number of agents. Existing trajectory predictions are fundamentally limited by lack of diversity in training data, which is difficult to acquire with sufficient coverage of possible modes. Our first contribution is an automatic method to simulate diverse trajectories in the top-view. It uses pre-existing datasets and maps as initialization, mines existing trajectories to represent realistic driving behaviors and uses a multi-agent vehicle dynamics simulator to generate diverse new trajectories that cover various modes and are consistent with scene layout constraints. Our second contribution is a novel method that generates diverse predictions while accounting for scene semantics and multi-agent interactions, with constant-time inference independent of the number of agents. We propose a convLSTM with novel state pooling operations and losses to predict scene-consistent states of multiple agents in a single forward pass, along with a CVAE for diversity. We validate our proposed multi-agent trajectory prediction approach by training and testing on the proposed simulated dataset and existing real datasets of traffic scenes. In both cases, our approach outperforms SOTA methods by a large margin, highlighting the benefits of both our diverse dataset simulation and constant-time diverse trajectory prediction methods.

Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction

“…Forecasting Methods: The future trajectory prediction has been investigated broadly in the literature using both classical [47,31,27] and deep learning based methods [18,1,44]. Deterministic models [1,33,41] predict most likely trajectory for each agent in the scene while neglecting the uncertainties inherited in the trajectory prediction problem.…”

“…To capture the uncertainties and create diverse trajectory predictions, stochastic methods have been proposed which encode possible modes of future trajectories through sampling random variables. Non-parametric deep generative models such as Conditional Variational Autoencoder (CVAE) [29,3,24,22,44] and Generative Adversarial Networks (GANs) [28,18,40] have been widely used in this domain. However, these methods fail to capture all underlying modes due to imbalance in the latent distribution [48].…”

“…Traditionally, generative models have been widely adapted to capture the uncertainties related to trajectory prediction problems [29,24,42,23,44]. However, generative methods may suffer from mode collapse issues, which reduces their applicability for safety critical applications such as self-driving cars.…”

“…Further, trajectory prediction methods incorporate driving knowledge using scene context either in the form of rasterized images [29,42,44,36,38,8] or by exploiting HD map data structure [32,17] as inputs. Usually, this information is represented as a feature given as input to the network and does not guarantee strong semantic coupling.…”

“…Specifically, we use a single representational model [44] for multi-agent inputs and encode interactions through novel use of hypercolumn descriptors [2] that extracts information from features at multiple scales. Moreover, we transform the prediction problem to normal-tangential (nt) coordinates with respect to input lanes.…”