RVNet: Deep Sensor Fusion of Monocular Camera and Radar for Image-Based Obstacle Detection in Challenging Environments

John, Vijay; Mita, Seiichi

doi:10.1007/978-3-030-34879-3_27

Cited by 78 publications

(54 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to [11], John et al [35] proposed a novel deep learning-based sensor fusion framework, known as the "RVNet". The RVNet was a single shot object detection network with two input branches and two output branches.…”

Section: Object Detection By Feature-level Fusionmentioning

confidence: 99%

“…Therefore, it is not a good idea to take the data level fusion scheme in the autonomous system for safety consideration. As for the feature-level fusion scheme [11,35,36], it is a recent popular fusion methodology. In general, the feature-level fusion scheme transforms radar points from the 3D world to a 2D image plane.…”

Section: Introductionmentioning

confidence: 99%

“…In addition, we do a lot of experiments to select the optimal hyperparameters in the radar image generation model, which is based on the improvement of detection performance. Inspired by the CNN-based feature fusion scheme, John et al [35] also proposed a CNN feature fusion detection model based on the YOLO [1] detection framework. More importantly, in the work of [35], the authors proved that the feature fusion scheme using radar and vision outperformed the other fusion schemes.…”

Section: Introductionmentioning

confidence: 99%

“…Inspired by the CNN-based feature fusion scheme, John et al [35] also proposed a CNN feature fusion detection model based on the YOLO [1] detection framework. More importantly, in the work of [35], the authors proved that the feature fusion scheme using radar and vision outperformed the other fusion schemes. In the work of [36], the authors proposed a CameraRadarFusionNet (CRF-Net) to learn at which level the fusion of the sensor data was the most beneficial for the detection task.…”

Section: Introductionmentioning

confidence: 99%

“…In contrast to the above schemes [11,35,36], in this paper, we propose a spatial attention fusion (SAF) method in the feature level, which can effectively merge the radar feature maps and vision feature maps. As we all know, radar points reflect the physical states of an autonomous vehicle's surrounding environment.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor

Chang

Zhang

et al. 2020

Sensors

108

View full text Add to dashboard Cite

For autonomous driving, it is important to detect obstacles in all scales accurately for safety consideration. In this paper, we propose a new spatial attention fusion (SAF) method for obstacle detection using mmWave radar and vision sensor, where the sparsity of radar points are considered in the proposed SAF. The proposed fusion method can be embedded in the feature-extraction stage, which leverages the features of mmWave radar and vision sensor effectively. Based on the SAF, an attention weight matrix is generated to fuse the vision features, which is different from the concatenation fusion and element-wise add fusion. Moreover, the proposed SAF can be trained by an end-to-end manner incorporated with the recent deep learning object detection framework. In addition, we build a generation model, which converts radar points to radar images for neural network training. Numerical results suggest that the newly developed fusion method achieves superior performance in public benchmarking. In addition, the source code will be released in the GitHub.

show abstract

Section: Object Detection By Feature-level Fusionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%