AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

Abstract

Predicting accurate future trajectories of multiple agents is essential for autonomous systems but is challenging due to the complex interaction between agents and the uncertainty in each agent's future behavior. Forecasting multiagent trajectories requires modeling two key dimensions:(1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. To this end, we propose a new Transformer, termed AgentFormer, that simultaneously models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. Extensive experiments show that our method significantly improves the state of the art on wellestablished pedestrian and autonomous driving datasets.

Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

“…In inference, we use greedy decoding to decode the future trajectory, and based on the experiments, using greedy decoding in training is able to increase the trajectory prediction accuracy during inference. This increase is also observed in [22]. Therefore, in both training and inference, the decoder decodes the future trajectory of the camera wearer in a greedy autoregressive manner.…”

“…STAR [21] predicts pedestrian trajectories with only the attention mechanism, which is achieved by a graph-based spatial transformer and a temporal transformer. AgentFormer [22], a transformer-based framework, jointly models temporal and social dimensions in human motion dynamics to predict future trajectories. Our model is also based on transformer, but we differ from them in that 1) we target for egocentric scenarios, and 2) our model encodes multiple modalities with a novel cascaded crossattention mechanism, whereas their models are designed to use the past trajectories as the only cue for the future trajectory prediction.…”

“…However, such data is currently lacking and limited research has been carried out in forecasting the trajectory of the camera wearer [9], [10], [11], [12]. Most human trajectory forecasting methods are targeted for scenarios captured from a bird-eye view or by a static camera [13], [14], [15], [16], [17], [18], [19], [20], [21], [22]. Different from these views, egocentric view brings up the opportunity of understanding how humans perceive the surroundings, and initiate/execute the responsive actions.…”

Vehicle Trajectory Prediction Using Generative Adversarial Network With Temporal Logic Syntax Tree Features

“…Recently, people have found transformers to have stronger encoding abilities and the advantage of avoiding recursion [13]. The current state-of-the-art algorithm of predicting human trajectories on ETH dataset, AgentFormer, uses transformer from end to end [14]. Therefore, we also experiment using transformer structure as encoder in addition to lstm to extract information from observed trajectories.…”

Delayed Propagation Transformer: A Universal Computation Engine towards Practical Control in Cyber-Physical Systems

“…In general, researchers use transformers to process the node embedding in two orthogonal directions: first, through the node-wise residual feature transformation, an arbitrary type of intra-node transformation is enabled [18,47,48]; second, through the attention mechanism, features from different nodes are dynamically aggregated and the inter-nodes relationships are captured [48]. Previous efforts have shown the potential of transformers in multi-agent system [49], by flattening connections features across time and agents.…”