GestureVLAD: Combining Unsupervised Features Representation and Spatio-Temporal Aggregation for Doppler-Radar Gesture Recognition

Abstract

In this paper we propose a novel framework to process Doppler-radar signals for hand gesture recognition. Doppler-radar sensors provide many advantages over other emerging sensing modalities, including low development costs and high sensitivity to capture subtle gestures with precision. Furthermore, they have attractive properties for ubiquitous deployment and can be conveniently embedded into different devices. In this scope, current recognition methods still rely in deep CNN-LSTM and 3D CNN-LSTM structures that require sufficient labelled data to optimize millions of parameters and significant amount of computational resources for inference; which limits their deployment. Indeed, subtle gestures recognition is a challenging task due to the high variability of gestures among different subjects. To overcome the challenges in the recognition task and the limitations of the current methods, we propose a shallow learning approach for gesture recognition, that is based on unsupervised range-Doppler features representation, along with a learnable pooling aggregation via NetVLAD. The proposed framework can encode extremely valuable information across time, and results in features that are highly discriminative for hand gesture recognition. Experimentation on publicly available Doppler-radar data shows that the proposed framework outperforms state-of-the-art approaches in terms of recognition accuracy and speed for sequence-level hand gesture classification.INDEX TERMS Convolutional neural networks, Doppler-radar, feature aggregation, hand gesture recognition, unsupervised representation learning.

Adaptive framework towards radar‐based diversity gesture recognition with range‐Doppler signatures

“…Similarly, the issue of degraded recognition performance also occurred in the study by Berenguer et al. [18].…”

“…To the best of our knowledge, there are only a few studies to explore the classifier's performance on unknown gesture sources, among which Ref. [16, 18] are the most representative ones. Both studies investigated this issue based on the publicly available Soli dataset, which is consistent with the dataset we used, and therefore the comparison is reasonable.…”

“…The public dataset is favoured by some researchers due to its reasonable experimental configurations [16, 18]. It includes 11 kinds of gestures and covers most of the common gestures oriented to tiny electronic devices, such as pinch index, finger rub, and so on.…”

Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds

“…In contrast, mmWave sensing features high bandwidth (4-7 GHz) and antenna apertures of few centimeters. For mmWave radars, gesture recognition is either model [41], [42] or data-driven [43], [44], [45], [46], [47], [48]. Most data-driven approaches combine Convolutional Neural Network (CNN) and RNN modules to process Doppler, range-Doppler, and/or angle-Doppler features [43], [44], [46].…”

“…For mmWave radars, gesture recognition is either model [41], [42] or data-driven [43], [44], [45], [46], [47], [48]. Most data-driven approaches combine Convolutional Neural Network (CNN) and RNN modules to process Doppler, range-Doppler, and/or angle-Doppler features [43], [44], [46]. Since these features are dependent on relative direction of movement and angle granularity, complex tasks, such as distinguishing simultaneous movement of different body parts becomes challenging.…”