Scene Transformer: A unified architecture for predicting multiple agent trajectories

Ngiam, Jiquan; Caine, Benjamin; Vasudevan, Vijay K.; Zhang, Zhengdong; Chiang, Hao-Tien Lewis; Ling, Jeffrey; Roelofs, Rebecca; Bewley, Alex; Liu, Chenxi; Venugopal, Ashish; Weiss, David J.; Sapp, Ben; Chen, Zhifeng; Shlens, Jonathon

doi:10.48550/arxiv.2106.08417

Cited by 10 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a fair comparison, we use the numbers reported on the official benchmark website [1] and only include the published models. Similar to the observations from the validation set, we observe that M2I improves mAP metrics by a large margin, compared to past WOMD interaction prediction challenge winners [27,38] and the existing state-of-the-art model [28].…”

Section: Testing Setsupporting

confidence: 75%

“…SceneTransformer [28] is a transformer-based model that leverages attention to combine features across road graphs and agent interactions both spatially and temporally. The model achieves state-of-the-art performance in the WOMD benchmark in both the marginal prediction task and the interactive prediction task.…”

Section: Quantitative Resultsmentioning

confidence: 99%

“…These hand-crafted functions require manual tuning and have difficulties modeling highly complicated and nonlinear interactions. In contrast, learning-based methods achieve better accuracy by learning interactions from realistic driving data: [2,16] utilize social pooling mechanisms to capture social influences from neighbor agents to predict interactive pedestrian trajectories in crowded scenes; [3,4,31] build a graph neural network (GNN) to learn the agent-toagent pairwise interactions; [24,28,29,34] leverage attention and transformer mechanisms to learn multi-agent interaction behaviors. In this work, we focus on predicting scene compliant trajectories through marginal models to afford better computational efficiency.…”

Section: Interactive Trajectory Predictionmentioning

confidence: 99%

“…We present the results in the interactive validation set in the top half of Tab. 1, where the baseline results are reported as in [10,28] mAP, the official ranking metric, over vehicles, and a better miss rate over pedestrians. Although M2I has higher minFDE errors, it has improved the mAP over all agents (the most right column) by a large margin, meaning our model generates a more accurate distribution with its confidence scores and outputs fewer false positive predictions.…”

Section: Validation Setmentioning

confidence: 99%

See 3 more Smart Citations

M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction

Sun¹,

Huang²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Predicting future motions of road participants is an important task for driving autonomously in urban scenes. Existing models excel at predicting marginal trajectories for single agents, yet it remains an open question to jointly predict scene compliant trajectories over multiple agents. The challenge is due to exponentially increasing prediction space as a function of the number of agents. In this work, we exploit the underlying relations between interacting agents and decouple the joint prediction problem into marginal prediction problems. Our proposed approach M2I first classifies interacting agents as pairs of influencers and reactors, and then leverages a marginal prediction model and a conditional prediction model to predict trajectories for the influencers and reactors, respectively. The predictions from interacting agents are combined and selected according to their joint likelihoods. Experiments show that our simple but effective approach achieves state-of-the-art performance on the Waymo Open Motion Dataset interactive prediction benchmark.

show abstract

Section: Testing Setsupporting

confidence: 75%

Section: Quantitative Resultsmentioning

confidence: 99%

Section: Interactive Trajectory Predictionmentioning

confidence: 99%

Section: Validation Setmentioning

confidence: 99%

See 2 more Smart Citations

M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction

Sun¹,

Huang²,

Gu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We use data augmentations to improve generalization ability of the model learned from the keypoint modality. Actor Dropout is performed by removing a random actor in a random frame, inspired by [68] that masks agents with probabilities to predict agent behaviors for autonomous driving. We remove actors by replacing the representation of the actor with a zero vector.…”

Section: Data Augmentationmentioning

confidence: 99%

COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality

Zhou¹,

Kadav²,

Shamsian³

et al. 2021

Preprint

View full text Add to dashboard Cite

Group Activity Recognition (GAR) detects the activity performed by a group of actors in a short video clip. The task requires the compositional understanding of scene entities and relational reasoning between them. We approach GAR by modeling the video as a series of tokens that represent the multi-scale semantic concepts in the video. We propose COMPOSER, a Multiscale Transformer based architecture that performs attention-based reasoning over tokens at each scale and learns group activity compositionally. In addition, we only use the keypoint modality which reduces scene biases and improves the generalization ability of the model. We improve the multi-scale representations in COMPOSER by clustering the intermediate scale representations, while maintaining consistent cluster assignments between scales. Finally, we use techniques such as auxiliary prediction and novel data augmentations (e.g., Actor Dropout) to aid model training. We demonstrate the model's strength and interpretability on the challenging Volleyball dataset. COMPOSER achieves a new state-of-the-art 94.5% accuracy with the keypoint-only modality. COMPOSER outperforms the latest GAR methods that rely on RGB signals, and performs favorably compared against methods that exploit multiple modalities. Our code will be available.

show abstract

Efficient policy detecting and reusing for non-stationarity in Markov games

Yan

Hao

Zhang

et al. 2020

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

This paper presents a Simple and effIcient Motion Prediction baseLine (SIMPL) for autonomous vehicles. Unlike conventional agent-centric methods with high accuracy but repetitive computations and scene-centric methods with compromised accuracy and generalizability, SIMPL delivers realtime, accurate motion predictions for all relevant traffic participants. To achieve improvements in both accuracy and inference speed, we propose a compact and efficient global feature fusion module that performs directed message passing in a symmetric manner, enabling the network to forecast future motion for all road users in a single feed-forward pass and mitigating accuracy loss caused by viewpoint shifting. Additionally, we investigate the continuous trajectory parameterization using Bernstein basis polynomials in trajectory decoding, allowing evaluations of states and their higher-order derivatives at any desired time point, which is valuable for downstream planning tasks. As a strong baseline, SIMPL exhibits highly competitive performance on Argoverse 1 & 2 motion forecasting benchmarks compared with other state-of-the-art methods. Furthermore, its lightweight design and low inference latency make SIMPL highly extensible and promising for real-world onboard deployment. We open-source the code at https: //github.com/HKUST-Aerial-Robotics/SIMPL.

show abstract

Scene Transformer: A unified architecture for predicting multiple agent trajectories

Cited by 10 publications

References 0 publications

M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction

M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction

COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality

Efficient policy detecting and reusing for non-stationarity in Markov games

Contact Info

Product

Resources

About