DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Li, Yingwei; Yu, Adams Wei; Meng, Tianjian; Caine, Ben; Ngiam, Jiquan; Peng, Daiyi; Shen, Junyang; Wu, Bo; Lu, Yifeng; Zhou, Denny; Le, Quoc V.; Yuille, Alan; Tan, Mingxing

doi:10.1109/cvpr52688.2022.01667

Cited by 197 publications

(71 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, the success of Transformer [10] draws much attention to Transformer-based LiDAR-camera fusion. Specifically, DeepFusion [11] fuses deep camera and LiDAR features instead of decorating raw LiDAR points at the input level. Therein, LearnableAlign is introduced that leverages the cross-attention mechanism to dynamically correlate Li-DAR information with the most related camera features.…”

Section: Fusion Methods For 3d Object Detectionmentioning

confidence: 99%

“…We also compare ImmFusion with the LiDAR-camera fusion methods, i.e. DeepFusion [11] and TokenFusion [12], which show the state-of-the-art accuracy in the task of 3D object detection. As DeepFusion is a generic Transformer-based block that is incompatible with the dimension-reduction mechanism, we adopt the parametric reconstruction pipeline by replacing the detection framework of DeepFusion with linear projection to regress SMPL-X parameters.…”

Section: E Comparison With Relevant Methodsmentioning

confidence: 99%

See 1 more Smart Citation

ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

Chen¹,

Wang²,

Shi³

et al. 2022

Preprint

View full text Add to dashboard Cite

3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementary, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for robust all-weather 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. In this paper, we present ImmFusion, the first mmWave-RGB fusion solution to reconstruct 3D human bodies in all weather conditions robustly. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments on a large-scale dataset, mmBody, captured in various environments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve a robust 3D human body reconstruction in all weather conditions. In addition, our method's accuracy is significantly superior to that of state-of-the-art Transformer-based LiDAR-camera fusion methods.

show abstract

Section: Fusion Methods For 3d Object Detectionmentioning

confidence: 99%

Section: E Comparison With Relevant Methodsmentioning

confidence: 99%

ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

Chen¹,

Wang²,

Shi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…YOLO [ 81 ] represented a big leap for object detection in 2D images, and 3D versions have been proposed [ 82 , 83 , 84 ]. Other approaches based on 3D descriptors [ 85 , 86 , 87 ] or other deep learning architectures [ 88 , 89 , 90 ] have also been the subject of research. These approaches do not rely on a priori knowledge about the objects to be recognized, but in human-made objects, it is usual that familiar geometric shapes are prevalent.…”

Section: Applicationsmentioning

confidence: 99%

RANSAC for Robotic Applications: A Survey

Martínez-Otzeta

Rodríguez-Moreno

Mendialdua

et al. 2022

Sensors

View full text Add to dashboard Cite

Random Sample Consensus, most commonly abbreviated as RANSAC, is a robust estimation method for the parameters of a model contaminated by a sizable percentage of outliers. In its simplest form, the process starts with a sampling of the minimum data needed to perform an estimation, followed by an evaluation of its adequacy, and further repetitions of this process until some stopping criterion is met. Multiple variants have been proposed in which this workflow is modified, typically tweaking one or several of these steps for improvements in computing time or the quality of the estimation of the parameters. RANSAC is widely applied in the field of robotics, for example, for finding geometric shapes (planes, cylinders, spheres, etc.) in cloud points or for estimating the best transformation between different camera views. In this paper, we present a review of the current state of the art of RANSAC family methods with a special interest in applications in robotics.

show abstract

“…The reasons are twofold: 1) Lidar scans provide comprehensive representations in 3D for inferring correspondences between sensors, and 2) camera images contain more semantic information to further boost the recognition ability. Various directions have been explored, such as image detection in 2D before projecting into frustums [32,54], two-stage frameworks with object-centric modality fusion [7,11,17], image feature-based lidar point decoration [50,51], or multi-level fusion [19,11,31]. Since sparse correspondences between camera and lidar are well defined, fusion is mostly focused on integrating information rather than matching points from different sensors.…”

Section: Introductionmentioning

confidence: 99%

CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection

Hwang¹,

Kretzschmar²,

Manela³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Robust 3D object detection is critical for safe autonomous driving. Camera and radar sensors are synergistic as they capture complementary information and work well under different environmental conditions. Fusing camera and radar data is challenging, however, as each of the sensors lacks information along a perpendicular axis, that is, depth is unknown to camera and elevation is unknown to radar. We propose the camera-radar matching network CramNet, an efficient approach to fuse the sensor readings from camera and radar in a joint 3D space. To leverage radar range measurements for better camera depth predictions, we propose a novel ray-constrained cross-attention mechanism that resolves the ambiguity in the geometric correspondences between camera features and radar features. Our method supports training with sensor modality dropout, which leads to robust 3D object detection, even when a camera or radar sensor suddenly malfunctions on a vehicle. We demonstrate the effectiveness of our fusion approach through extensive experiments on the RADIATE dataset, one of the few large-scale datasets that provide radar radio frequency imagery. A camera-only variant of our method achieves competitive performance in monocular 3D object detection on the Waymo Open Dataset.

show abstract

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

Cited by 197 publications

References 25 publications

ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

RANSAC for Robotic Applications: A Survey

CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection

Contact Info

Product

Resources

About