Review on Millimeter-Wave Radar and Camera Fusion Technology

Zhou, Yong; Dong, Yanyan; Hou, Fujin; Wu, Jianqing

doi:10.3390/su14095114

Cited by 21 publications

(6 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this step, the aligned features from vision and radar modalities are fused to leverage their complementary information [12]. Various fusion techniques can be applied, including early fusion, late [13]. In this approach, multimodal fusion is employed to combine the decision results from the vision module with the raw velocity data from FMCW, facilitating comprehensive gesture recognition.…”

Section: Feature Fusionmentioning

confidence: 99%

Real-time gesture-based control of UAVs using multimodal fusion of FMCW radar and vision

Yang

2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Gesture-based control has gained prominence as an intuitive and natural means of interaction with unmanned aerial vehicles (UAVs). This paper presents a real-time gesture-based control system for UAVs that leverages the multimodal fusion of Frequency Modulated Continuous Wave (FMCW) radar and vision sensors, aiming to enhance user experience through precise and responsive UAV control via hand gestures. The research focuses on developing an effective fusion framework that combines the complementary advantages of FMCW radar and vision sensors. FMCW radar provides robust range and velocity measurements, while vision sensors capture fine-grained visual information. By integrating data from these modalities, the system achieves a comprehensive understanding of hand gestures, resulting in improved gesture recognition accuracy and robustness. The proposed system comprises three main stages: data acquisition, gesture recognition, and multimodal fusion. In the data acquisition stage, synchronized data streams from FMCW radar and vision sensors are captured. Then, machine learning algorithms are employed in the gesture recognition stage to classify and interpret hand gestures. Finally, the multimodal fusion stage aligns and fuses the data, creating a unified representation that captures the spatial and temporal aspects of hand gestures, enabling real-time control commands for the UAV. Experimental results demonstrate the system‘s effectiveness in accurately recognizing and responding to hand gestures. The multimodal fusion of FMCW radar and vision sensors enables a robust and versatile gesture-based control interface.

show abstract

Section: Feature Fusionmentioning

confidence: 99%

Real-time gesture-based control of UAVs using multimodal fusion of FMCW radar and vision

Yang

2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…The Kalman filtering technique [ 32 ] was used to fuse the objects’ positions, which is accurate in estimating a global position. This method has been used in several successful works for this purpose [ 15 ]. Since the detected object position on the sensor is more accurate than the camera, we related the two coordinates to obtain an accurate result.…”

Section: Proposed Frameworkmentioning

confidence: 99%

“…Also, using a mmWave sensor can notify the system of the presence of a person to start the facial recognition process. Further details on existing methods for integrating mmWave radar and camera can be found in [ 13 , 14 , 15 ]. The review papers present all the challenges associated with integration, such as synchronization and tracking multiple people.…”

Section: Introductionmentioning

confidence: 99%

A New Framework for Smart Doors Using mmWave Radar and Camera-Based Face Detection and Recognition Techniques

Akbari,

Al-Binali,

Al-Mohannadi

et al. 2023

Sensors

View full text Add to dashboard Cite

By integrating IoT technology, smart door locks can provide greater convenience, security, and remote access. This paper presents a novel framework for smart doors that combines face detection and recognition techniques based on mmWave radar and camera sensors. The proposed framework aims to improve the accuracy and some security aspects arising from some limitations of the camera, such as overlapping and lighting conditions. By integrating mmWave radar and camera-based face detection and recognition algorithms, the system can accurately detect and identify people approaching the door, providing seamless and secure access. This framework includes four key components: person detection based on mmWave radar, camera preparation and integration, person identification, and door lock control. The experiments show that the framework can be useful for a smart home.

show abstract

“…Radar and camera data can be mixed at mid-level [78]. For example, it is possible to use the radar to detect the presence of a target in a particular area and then the visual data from the camera can be used to recognize the object in the surrounding of the detected target, avoiding the exploration of the entire image.…”

Section: Sensor Fusionmentioning

confidence: 99%

A Survey of Automotive Radar and Lidar Signal Processing and Architectures

Giuffrida,

Masera,

Martina

2023

Chips

View full text Add to dashboard Cite

In recent years, the development of Advanced Driver-Assistance Systems (ADASs) is driving the need for more reliable and precise on-vehicle sensing. Radar and lidar are crucial in this framework, since they allow sensing of vehicle’s surroundings. In such a scenario, it is necessary to master these sensing systems, and knowing their similarities and differences is important. Due to ADAS’s intrinsic real-time performance requirements, it is almost mandatory to be aware of the processing algorithms required by radar and lidar to understand what can be optimized and what actions can be taken to approach the real-time requirement. This review aims to present state-of-the-art radar and lidar technology, mainly focusing on modulation schemes and imaging systems, highlighting their weaknesses and strengths. Then, an overview of the sensor data processing algorithms is provided, with some considerations on what type of algorithms can be accelerated in hardware, pointing to some implementations from the literature. In conclusion, the basic concepts of sensor fusion are presented, and a comparison between radar and lidar is performed.

show abstract

Review on Millimeter-Wave Radar and Camera Fusion Technology

Cited by 21 publications

References 79 publications

Real-time gesture-based control of UAVs using multimodal fusion of FMCW radar and vision

Real-time gesture-based control of UAVs using multimodal fusion of FMCW radar and vision

A New Framework for Smart Doors Using mmWave Radar and Camera-Based Face Detection and Recognition Techniques

A Survey of Automotive Radar and Lidar Signal Processing and Architectures

Contact Info

Product

Resources

About