UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View

Qin, Zequn; Chen, Jingyu; Chen, Chao; Chen, Xiaozhi; Li, Xi

doi:10.48550/arxiv.2207.08536

Cited by 2 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BEVDet4D [64] fuses the previous feature maps with the current frame using a spatial alignment operation followed by a concatenation of multiple feature maps. BEV-Former [4] and UniFormer [126] adopt a soft way to fusion temporal information. The attention module is utilized to fuse temporal information from previous BEV feature maps and previous frames, respectively.…”

Section: Temporal Fusionmentioning

confidence: 99%

Delving Into the Devils of Bird’s-Eye-View Perception: A Review, Evaluation and Recipe

Li,

Sima,

Dai

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Learning powerful representations in bird's-eye-view (BEV) for perception tasks is trending and drawing extensive attention both from industry and academia. Conventional approaches for most autonomous driving algorithms perform detection, segmentation, tracking, etc., in a front or perspective view. As sensor configurations get more complex, integrating multi-source information from different sensors and representing features in a unified view come of vital importance. BEV perception inherits several advantages, as representing surrounding scenes in BEV is intuitive and fusion-friendly; and representing objects in BEV is most desirable for subsequent modules as in planning and/or control. The core problems for BEV perception lie in (a) how to reconstruct the lost 3D information via view transformation from perspective view to BEV; (b) how to acquire ground truth annotations in BEV grid; (c) how to formulate the pipeline to incorporate features from different sources and views; and (d) how to adapt and generalize algorithms as sensor configurations vary across different scenarios. In this survey, we review the most recent works on BEV perception and provide an in-depth analysis of different solutions. Moreover, several systematic designs of BEV approach from the industry are depicted as well. Furthermore, we introduce a full suite of practical guidebook to improve the performance of BEV perception tasks, including camera, LiDAR and fusion inputs. At last, we point out the future research directions in this area. We hope this report will shed some light on the community and encourage more research effort on BEV perception. We keep an active repository to collect the most recent work and provide a toolbox for bag of tricks at https://github.com/OpenDriveLab/Birds-eye-view-Perception.

show abstract

Section: Temporal Fusionmentioning

confidence: 99%

Delving Into the Devils of Bird’s-Eye-View Perception: A Review, Evaluation and Recipe

Li,

Sima,

Dai

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…PETR and BEVFormer utilize the end-to-end 3D detection head in DETR [6], treating all BEV grids as dense queries. For detecting occluded and dynamic objects better, multi-frame temporal information is also introduced into the origin detection architecture [15,26,34].…”

Section: Related Workmentioning

confidence: 99%

Multi-Camera Calibration Free BEV Representation for 3D Object Detection

Jiang¹,

Meng²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In advanced paradigms of autonomous driving, learning Bird's Eye View (BEV) representation from surrounding views is crucial for multi-task framework. However, existing methods based on depth estimation or camera-driven attention are not stable to obtain transformation under noisy camera parameters, mainly with two challenges, accurate depth prediction and calibration. In this work, we present a completely Multi-Camera Calibration Free Transformer (CFT) for robust BEV representation, which focuses on exploring implicit mapping, not relied on camera intrinsics and extrinsics. To guide better feature learning from image views to BEV, CFT mines potential 3D information in BEV via our designed position-aware enhancement (PA). Instead of camera-driven point-wise or global transformation, for interaction within more effective region and lower computation cost, we propose a view-aware attention which also reduces redundant computation and promotes converge. CFT achieves 49.7% NDS on the nuScenes detection task leaderboard, which is the first work removing camera parameters, comparable to other geometry-guided methods. Without temporal input and other modal information, CFT achieves second highest performance with a smaller image input (1600 × 640). Thanks to view-attention variant, CFT reduces memory and transformer FLOPs for vanilla attention by about 12% and 60%, respectively, with improved NDS by 1.0%. Moreover, its natural robustness to noisy camera parameters makes CFT more competitive.

show abstract

UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View

Cited by 2 publications

References 20 publications

Delving Into the Devils of Bird’s-Eye-View Perception: A Review, Evaluation and Recipe

Delving Into the Devils of Bird’s-Eye-View Perception: A Review, Evaluation and Recipe

Multi-Camera Calibration Free BEV Representation for 3D Object Detection

Contact Info

Product

Resources

About