SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

Abstract

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with a physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.

Scene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids

“…Social GAN [8] Sophie [10] Desire [12] Ours Linear Regressor Social Forces [33] Social LSTM [5] CAR-NET [ Fig. 4.…”

“…In addition to location co-ordinates, some approaches also incorporate auxiliary information such as the head pose of pedestrians [9], [14] while encoding past motion. Many approaches jointly model the past motion of multiple agents in the scene to capture interaction between agents [5], [15], [12], [10], [7], [11]. This is typically done by pooling the RNN states of individual agents in a social tensor [5], [12], [11], using graph neural networks [16] or by modeling pairwise distances between agents along with max pooling [8], [10], [7].…”

Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

“…Social GAN [35] defines a spatial pooling for motion prediction. SoPhie [18] introduces an attentive GAN to predict individual trajectories leveraging the physical Constraints. Although obtained impressive results, these top-down methods pose limitations that make them inapplicable to egocentric applications of selfdriving scenarios.…”

“…Although they have obtained convincing performance on several benchmarks, completion of such tasks is not enough for human-like driving. On the other hand, trajectory prediction [11]- [18] addresses the problem to some extent by predicting the potential future position of the pedestrian. But predicting trajectories with high confidence long-enough into the future is a very challenging task as many different and subtle factors change the trajectories of pedestrians.…”

Effects of a Social Force Model Reward in Robot Navigation Based on Deep Reinforcement Learning

“…For example, sequence models that use Long Short Term Memory (LSTM) recurrent neural networks like Social LSTM [1] and other [12,13] are capable to encode the Human-Robot interactions and Human-Human interactions to improve the predictions. Other techniques are based in generative models like Social-GAN [11] or SoPhie [21] that use pooling modules and attention modules. This type of techniques offers very useful information in navigation tasks but not take into account all the elements in navigation tasks like obstacles, actions, kinematics or goals.…”