ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries

Abstract

Existing autonomous driving pipelines separate the perception module from the prediction module. The two modules communicate via hand-picked features such as agent boxes and trajectories as interfaces. Due to this separation, the prediction module only receives partial information from the perception module. Even worse, errors from the perception modules can propagate and accumulate, adversely affecting the prediction results. In this work, we propose ViP3D, a visual trajectory prediction pipeline that leverages the rich information from raw videos to predict future trajectories of agents in a scene. ViP3D employs sparse agent queries throughout the pipeline, making it fully differentiable and interpretable. Furthermore, we propose an evaluation metric for this novel end-to-end visual trajectory prediction task. Extensive experimental results on the nuScenes dataset show the strong performance of ViP3D over traditional pipelines and previous end-to-end models. 1

Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking

“…With a lower ADE and FDE, PF-Track has better trajectory quality. Our conclusions agree with previous studies [17,38,62]. More specifically, LSTM is a shallow model, unable to capture meaningful dynamics from noisy tracks; the stronger VectorNet can perform better than the other baselines but it is still worse than forecasting trajectories in an end-to-end framework, as proposed in our method.…”

“…In the prediction literature, on the other hand, it is common to assume the availability of ground truth object trajectories and HD maps [3,7,12,66]. A few attempts for a more realistic evaluation have been made [17,22], focusing only on the prediction performance.…”

GTOC 11: Results from Tsinghua University and Shanghai Institute of Satellite Engineering

“…Recent methods follow this concept to output planning results for the ego car given sensor inputs [17,18,25,53,63]. Most of them follow a conventional pipeline of perception [21,32,33,57,65], prediction [11,14,36,66], and planning [22,23,54,67]. They usually first perform BEV perception to extract relevant information (e.g., 3D agent boxes, semantic maps, tracklets) and then exploit them to infer future trajectories of agents and the ego vehicle.…”

“…The following methods incorporated more data [63] or extracted more intermediate features [17,18,25] to provide more information for the planner, which achieved remarkable performance. Most methods only model object motions and cannot capture the fine-grained structural and semantic information of the surroundings [11,14,24,25,66]. Differently, we propose a world model to predict the evolution of both the surrounding dynamic and static elements.…”

“…Perception aims to obtain a semantic understanding of the surrounding scene such as 3D object detection [19,32,33] and semantic map construction [30,34,37,66]. The subsequent prediction module captures the motion of other traffic participants [11,14,24,66], and the planning module then makes decisions based on previous outputs [17,18,25,45]. How-ever, this serial design usually requires ground-truth labels at each stage of training, yet the instance-level bounding boxes and high-definition maps are difficult to annotate.…”