On the Effect of Training Convolution Neural Network for Millimeter-Wave Radar-Based Hand Gesture Recognition

Abstract

The purpose of this paper was to investigate the effect of a training state-of-the-art convolution neural network (CNN) for millimeter-wave radar-based hand gesture recognition (MR-HGR). Focusing on the small training dataset problem in MR-HGR, this paper first proposed to transfer the knowledge with the CNN models in computer vision to MR-HGR by fine-tuning the models with radar data samples. Meanwhile, for the different data modality in MR-HGR, a parameterized representation of temporal space-velocity (TSV) spectrogram was proposed as an integrated data modality of the time-evolving hand gesture features in the radar echo signals. The TSV spectrograms representing six common gestures in human–computer interaction (HCI) from nine volunteers were used as the data samples in the experiment. The evaluated models included ResNet with 50, 101, and 152 layers, DenseNet with 121, 161 and 169 layers, as well as light-weight MobileNet V2 and ShuffleNet V2, mostly proposed by many latest publications. In the experiment, not only self-testing (ST), but also more persuasive cross-testing (CT), were implemented to evaluate whether the fine-tuned models generalize to the radar data samples. The CT results show that the best fine-tuned models can reach to an average accuracy higher than 93% with a comparable ST average accuracy almost 100%. Moreover, in order to alleviate the problem caused by private gesture habits, an auxiliary test was performed by augmenting four shots of the gestures with the heaviest misclassifications into the training set. This enriching test is similar with the scenario that a tablet reacts to a new user. The results of two different volunteer in the enriching test shows that the average accuracy of the enriched gesture can be improved from 55.59% and 65.58% to 90.66% and 95.95% respectively. Compared with some baseline work in MR-HGR, the investigation by this paper can be beneficial in promoting MR-HGR in future industry applications and consumer electronic design.

Wi-GC: A Deep Spatiotemporal Gesture Recognition Method Based on Wi-Fi Signal

“…Gesture recognition based on millimeter-wave radar uses millimeter-level signals sent and received by radar equipment and processes and recognizes them. For example, Zhang et al [7] use millimeter-wave radar signals for gesture recognition and convert millimeter-wave data into a Time and Space Velocity (TSV) spectrogram. Then the gesture features are extracted by a specific feature extraction algorithm and classified with a custom classifier, achieving an accuracy of 93%.…”

“…It supports many emerging Internet of Things (IoT) applications such as user recognition [2], smart home [3], healthcare [4], etc. Generally, the technologies based on gesture recognition include sensors [5], web cameras [6], and millimeter-wave radars [7]. However, they all have certain limitations.…”

Multi-Sensor-Based Blind-Spot Reduction Technology and a Data-Logging Method Using a Gesture Recognition Algorithm Based on Micro E-Mobility in an IoT Environment

“…Autonomous driving is defined as driving technology that allows a vehicle to travel to a destination on its own, without the need for a driver to operate it [ 5 ]. As such, autonomous driving is evolving from existing automobile technology to a new type of convergence technology through the convergence of automobiles and various sensors (e.g., cameras and LIDAR) that became universalized between the first industrial revolution and the third industrial revolution [ 6 ].…”

Foot Gesture Recognition Using High-Compression Radar Signature Image and Deep Learning

“…Existing sensors for a gesture recognition of movements of hands and arms include ultrasound [ 9 , 10 , 11 , 12 ], camera based vision [ 13 , 14 , 15 , 16 , 17 ], and radar [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ]. Ultrasonic sensors have the advantage of relatively low price, but they have short detection distance.…”