Extending Reliability of mmWave Radar Tracking and Detection via Fusion With Camera

Abstract

In this paper, a new radar-camera fusion system is presented. The fusion system takes into consideration the error bounds of the two different coordinate systems from the heterogeneous sensors, and further a new fusion-extended Kalman filter is utilized to adapt to the heterogeneous sensors. Real-world application considerations such as asynchronous sensors, multi-target tracking and association are also studied and illustrated in this paper. Experimental results demonstrated that the proposed fusion system can realize a range accuracy of 0.29m with an angular accuracy of 0.013rad in real-time. Therefore, the proposed fusion system is effective, reliable and computationally efficient for real-time kinematic fusion applications.

Millimeter Wave Sensing: A Review of Application Pipelines and Building Blocks

“…Noise reduction with help from Singular Value Decomposition (SVD) can be applied to extract desired spectrogram parts indicating presence of target reflection [142]. A feed-forward neural network can be used to filter ghost targets (i.e., superimposed noise from unwanted background and target reflections) in automotive radar sensing [95]. The bandpass and highpass filters reported in this section are applied in the frequency domain.…”

“…Several Bayesian filters exist, such as the custom Bayesian [45], [185], particle [44], [136], α − β [128], Kalman [46], [53], [58], extended Kalman [55], [94], [96], [114], [115], [135], [186], fusion extended Kalman [95], [146], unscented Kalman [89], fusion adaptive Kalman [51], and adaptive Sage-Husa Kalman [52], [61] filter. The Kalman filter [219], under linear, quadratic, and Gaussian assumptions, can represent the state transition and observation functions x t = g(x t−1 , pn t ) and s t = h(x t , mn t ) as a set of linear equations.…”

“…There are two versions of tree-based hybrid models as addressed in [95], [128], [136], [186]. In the first version, the tree-based hybrid model either branches off to perform multiple processes simultaneously or fuses the output of several simultaneously performed processes to perform a single consecutive process.…”

“…In the first version, the tree-based hybrid model either branches off to perform multiple processes simultaneously or fuses the output of several simultaneously performed processes to perform a single consecutive process. In addition to vision data fusion with a Bayesian filter, millimeter wave data is first clustered [95]. Simultaneous Localization And Mapping (SLAM) is solved by first clustering a raw radar scan to make it more sparse.…”

“…Some pipelines apply sensor fusion of millimeter wave data with data originating from other sensing domains. Sensing domains include vision [51], [61], [66], [67], [71], [95], [105], [120], [146], depth [44], [105], lidar [71], inertial measurements [89], and exerted forces [91]. Data originating from these domains complement millimeter wave data and therefore cause more accurate analytical modeling performance in several situations.…”