RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization

Abstract

Various autonomous or assisted driving strategies have been facilitated through the accurate and reliable perception of the environment around a vehicle. Among the commonly used sensors, radar has usually been considered as a robust and cost-effective solution even in adverse driving scenarios, e.g., weak/strong lighting or bad weather. Instead of considering to fuse the unreliable information from all available sensors, perception from pure radar data becomes a valuable alternative that is worth exploring. However, unlike rich RGB-based images captured by a camera, it is noticeably difficult to extract semantic information from the radar signals. In this paper, we propose a deep radar object detection network, named RODNet, which is cross-supervised by a camera-radar fused algorithm without laborious annotation efforts, to effectively detect objects from the radio frequency (RF) images in real-time. First, the raw signals captured by millimeter-wave radars are transformed to RF images in range-azimuth coordinates. Second, our proposed RODNet takes a sequence of RF images as the input to predict the likelihood of objects in the radar field of view (FoV). Two customized modules are also added to handle multi-chirp information and object relative motion. Instead of using humanlabeled ground truth for training, the proposed RODNet is crosssupervised by a novel 3D localization of detected objects using a camera-radar fusion (CRF) strategy in the training stage. Finally, we propose a method to evaluate the object detection performance of the RODNet. Due to no existing public dataset available for our task, we create a new dataset, named CRUW 1 , which contains synchronized RGB and RF image sequences in various driving scenarios. With intensive experiments, our proposed cross-supervised RODNet achieves 86% average precision and 88% average recall of object detection performance, which shows the robustness to noisy scenarios in various driving conditions.

Radar Transformer: An Object Classification Network Based on 4D MMW Imaging Radar

“…As we know, deep learning is data-driven, and quality data are crucial for neural network training. At present, the public datasets containing radar information mainly include Nuscence [ 35 ], CRUW [ 36 ] and Oxford Radar Robotcar Dataset [ 37 ]. However, these datasets contain only 2D radar point information or radiofrequency (RF) images of the radar.…”

Efficient-ROD

“…Afterward, conventional 3D transposed convolution and an extra 3D convolutional operation are performed to decode the spatial information from the embedded Spatio-temporal feature Z 3 + Z 3 . Finally, a neck layer is used to project the decoded Spatio-temporal feature to the confidence map with shape × × × × , and followed by applying L-NMS in [15] to retrieve the detection results.…”

“…In this manner, the radar object detection can be performed via a single neural network without two-stage or two data sources in the inference phase. In [15], the temporal deformable convolution (TDC) was further proposed to capture the radar features in time effectively. In this way, the performance can be boosted further without increasing inference time significantly.…”

“…However, the claim in [14,15] is real-time radar object detection, is hard to meet when the architecture is fully functional (i.e., the best performance of the model, HwGI in [14], and HG with TDC in [15]. Since the larger kernel size was adopted in the HwGI model of detection network to obtain larger spatial and temporal receptive fields, the computational complexity will be proportionally increased.…”

DANet

“…There is only a few researches [12,16,17] about how to utilize the rich semantic information in the radar signal for the object detection task. Although those related researches have explored the object detection task on RF data, there is no public benchmark in this area until that piece of work [26] exists, to the best of our knowledge. The most important innovation of that paper is it proposed a cross-model supervision pipeline for annotating object labels on the radio frequency (RF) images by a camera-radar fused (CRF) algorithm automatically and established a public benchmark for radar object detection task.…”