3D-MAN: 3D Multi-frame Attention Network for Object Detection

Yang, Zetong; Zhou, Yin; Chen, Zhifeng; Ngiam, Jiquan

doi:10.1109/cvpr46437.2021.00190

Cited by 79 publications

(65 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These 2D pillars can then be processed with existing 2D convolutional detection networks to produce the bird-eye-view bounding boxes. Since 2D pillars are usually easy and fast to process, many recent 3D detection methods [34,38,43,44] are built upon PointPillars. In this paper, we also choose PointPillar as our baseline approach for dealing with lidar point clouds.…”

Section: Related Workmentioning

confidence: 99%

“…3D detection models. We reimplement three popular point cloud 3D object detection methods, PointPillars [16], Cen-terPoint [44], and 3D-MAN [43], as baselines. Besides, we also find that their improved versions (denote as Point-Pillars++, CenterPoint++, and 3D-MAN++) can be better baselines, which use 3 layers of multilayer perceptron with hidden size 256 to construct pseudo image from point cloud inputs, and change the non-linear activation function from ReLU [9,21] to SILU [7,28].…”

Section: Implementation Detailsmentioning

confidence: 99%

“…Point cloud 3D object detection methods. We reimplement three popular point cloud 3D object detection methods, PointPillars [16], CenterPoint [44], and 3D-MAN [43]. As mentioned in Section 2, PointPillars voxelize the point cloud by pillars, a single tall elongated voxel per map location, to construct bird eye view pseudo image; finally, the pseudo image is fed to an anchor-based object detection pipeline.…”

Section: A2 Implementation Details Of 3d Detectorsmentioning

confidence: 99%

“…Note that we only implemented the PointPillars-based single-stage version of CenterPoint. 3D-MAN is similar to CenterPoint, and the main difference is when computing the loss, 3D-MAN uses a Hungarian algorithm to associate the prediction and the ground-truth (See Section 3.1 of Yang et al [43] for more details). Our improved implementations.…”

Section: A2 Implementation Details Of 3d Detectorsmentioning

confidence: 99%

“…On the Waymo Open Dataset, DeepFusion improves several prevalent 3D detection models such as PointPillars [16], CenterPoints [44], and 3D-MAN [43] by 6.7, 8.9, and 6.2 LEVEL 2 APH, respectively. We achieve state-ofthe-art results on Waymo Open Dataset that DeepFusion improves 7.4 Pedestrian LEVEL 2 APH over PointAugmenting [36], the previous best multi-modal method, on the validation set.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Li¹,

Yu²,

Meng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods [34,36] simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and Learn-ableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves PointPillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7, 8.9, and 6.2 LEVEL 2 APH, respectively. Notably, our models achieve state-ofthe-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo/ tree/master/lingvo.

show abstract

Section: Related Workmentioning

confidence: 99%