Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor

Abstract

For autonomous driving, it is important to detect obstacles in all scales accurately for safety consideration. In this paper, we propose a new spatial attention fusion (SAF) method for obstacle detection using mmWave radar and vision sensor, where the sparsity of radar points are considered in the proposed SAF. The proposed fusion method can be embedded in the feature-extraction stage, which leverages the features of mmWave radar and vision sensor effectively. Based on the SAF, an attention weight matrix is generated to fuse the vision features, which is different from the concatenation fusion and element-wise add fusion. Moreover, the proposed SAF can be trained by an end-to-end manner incorporated with the recent deep learning object detection framework. In addition, we build a generation model, which converts radar points to radar images for neural network training. Numerical results suggest that the newly developed fusion method achieves superior performance in public benchmarking. In addition, the source code will be released in the GitHub.

MMW Radar-Based Technologies in Autonomous Driving: A Review

“…These algorithms project radar data to image coordinate system. Moreover, as it is known that radar data is sparse compared with image, these algorithms try to make the best of different dimension information of radar measurement, such as distance, speed, and intensity to fill multiple channels of “radar sparse image” [ 26 ]. Multi-frame data is also used here to increase the density of radar data [ 25 ].…”

“…Thirdly to address the unbalance between positive and negative samples, focal loss is adopted to design the loss function. [ 26 , 75 ]. Under multifaceted efforts, deep fusion networks reveal their good performance as is showed In Table 2 .…”

“…Specific convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are proposed for radar data processing [ 23 , 24 ]. Furthermore, related studies use DNN to improve the fusion performance [ 25 , 26 ]. To satisfy the enormous demands of radar data In deep neural network (DNN) training, new datasets carrying the full autonomous vehicle sensor suite of radar, LiDAR, and visual sensors such as nuScenes are published [ 27 ].…”

High-Resolution Neural Network for Driver Visual Attention Prediction

“…Recently, the studies of predicting a driver's attention locations have shown dramatic progress, owing to the development of deep convolutional neural network frameworks [10][11][12][13]. Existing attention prediction approaches based on deep neural networks leverage semantic segmentation networks that predict a class label among a set of predefined class labels for each pixel of an image.…”

A New Roadway Eventual Obstacle Detection System Based on Computer Vision

“…Other authors have developed solutions based on machine learning (ML) or deep learning (DL): Algabri and Choi, [ 28 ] present a method to detect and track people in indoor environments, whereas Qiu et al [ 29 ] focused on different types of moving obstacles in outdoor environments. More recently, other authors, Chang et al [ 30 ], Yang et al [ 31 ] and Qiu et al [ 32 ] have developed solutions for obstacles detection for self-driving cars. However, the main drawback of these solutions based on ML/DL is that, most of them, require a lot of training and thus existing data.…”