M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers

Abstract

We present a novel architecture for 3D object detection, M3DETR, which combines different point cloud representations (raw, voxels, bird-eye view) with different feature scales based on multi-scale feature pyramids. M3DETR is the first approach that unifies multiple point cloud representations, feature scales, as well as models mutual relationships between point clouds simultaneously using transformers. We perform extensive ablation experiments that highlight the benefits of fusing representation and scale, and modeling the relationships. Our method achieves state-of-the-art performance on the KITTI 3D object detection dataset and Waymo Open Dataset. Results show that M3DETR improves the baseline significantly by 1.48% mAP for all classes on Waymo Open Dataset. In particular, our approach ranks 1 st on the well-known KITTI 3D Detection Benchmark for both car and cyclist classes, and ranks 1 st on Waymo Open Dataset with single frame point cloud input.

Multimodal Virtual Point 3D Detection

“…Secondly, our current two-stage refinement modules only use features from the bird-eye view which may not take full advantage of the high-resolution virtual points generated from our algorithm. We believe point or voxel-based two-stage 3D detectors like PVRCNN [45] and M3Detr [15] may give more significant improvements. Finally, the point-based abstraction connecting 2D and 3D detection may introduce too large of a bottleneck to transmit information from 2D to 3D.…”

LCPFormer: Towards Effective 3D Point Cloud Analysis via Local Context Propagation in Transformers

“…Alternatively, [44] PSNet proposes a fast data structuring method to tackle the data structuring issue in point-based methods. There are also other methods that combine both point-wise and voxel-wise features in a local field [10], [45] or a global field [46]- [48]. In addition, some of these approaches propose using continuous convolution on the 3D point clouds by the designed kernel structure.…”

Slice Transformer and Self-supervised Learning for 6DoF Localization in 3D Point Cloud Maps

“…Consequently, Transformer architectures have also started to emerge for point cloud processing. For instance, [24] and [25] employ Transformers for 3D object detection, whereas [26] developed a method for point cloud segmentation. However, due to the complex nature of outdoor LiDAR data, Transformer models are yet to establish themselves on outdoor benchmarks.…”