Imagining The Road Ahead: Multi-Agent Trajectory Prediction via Differentiable Simulation

Abstract

We develop a deep generative model built on a fully differentiable simulator for multi-agent trajectory prediction. Agents are modeled with conditional recurrent variational neural networks (CVRNNs), which take as input an ego-centric birdview image representing the current state of the world and output an action, consisting of steering and acceleration, which is used to derive the subsequent agent state using a kinematic bicycle model. The full simulation state is then differentiably rendered for each agent, initiating the next time step. We achieve state-of-the-art results on the INTERACTION dataset, using standard neural architectures and a standard variational training objective, producing realistic multi-modal predictions without any ad-hoc diversity-inducing losses. We conduct ablation studies to examine individual components of the simulator, finding that both the kinematic bicycle model and the continuous feedback from the birdview image are crucial for achieving this level of performance. We name our model ITRA, for "Imagining the Road Ahead".

StarNet: Joint Action-Space Prediction with Star Graphs and Implicit Global Frame Self-Attention

“…Hence, virtually all recent state-of-the-art approaches use graph-based learned models such as GNNs and Multi-Head Attention (MHA), which is related to the Transformer architecture 2 [6]. For the sake of brevity, we limit our review to joint prediction works [12], [18], [33], [2], [3], [4], [34], [35], [20], [5].…”

“…INTERACTION [7] ADE FDE DESIRE [40] 0.32 0.88 MultiPath [15] 0.30 0.99 STG-DAT [34] 0.29 0.54 TNT [16] 0.21 0.67 ReCoG [41] 0.19 0.66 HEAT-I-R [35] 0.19 0.65 ITRA [5] 0.17 ). The values for [40] and [15] are given in [16].…”

“…In all architectures, we set the number of predicted modes m to 3. As metrics, we computed the Average Displacement Error (ADE) and Final Displacement Error (FDE), defined as in [5].…”

“…In handling the joint nature of the prediction problem, many works use global representations centered around the autonomous vehicle (AV) [1], [2], [3], [4], [5], in the form of sensor data or generated Bird's Eye View (BEV) images. In order to obtain agent-specific features, they extract patches around the agents of interest, derive corresponding latent information, and perform individual or joint predictions, provided that the latent features are combined.…”