RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training Examples

Abstract

Microwave radars bring many benefits to mid-air gesture sensing due to their large field of view and independence from environmental conditions, such as ambient light and occlusion. However, radar signals are highly dimensional and usually require complex deep learning approaches. To understand this landscape, we report results from a systematic literature review of ( N = 118) scientific papers on radar sensing, unveiling a large variety of radar technology of different operating frequencies and bandwidths, antenna configurations, but also various gesture recognition techniques. Although highly accurate, these techniques require a large amount of training data that depend on the type of radar. Therefore, the training results cannot be easily transferred to other radars. To address this aspect, we introduce a new gesture recognition pipeline that implements advanced full-wave electromagnetic modeling and inversion to retrieve physical characteristics of gestures that are radar independent, i.e. , independent of the source, antennas, and radar-hand interactions. Inversion of radar signals further reduces the size of the dataset by several orders of magnitude, while preserving the essential information. This approach is compatible with conventional gesture recognizers, such as those based on template matching, which only need a few training examples to deliver high recognition accuracy rates. To evaluate our gesture recognition pipeline, we conducted user-dependent and user-independent evaluations on a dataset of 16 gesture types collected with the Walabot, a low-cost off-the-shelf array radar. We contrast these results with those obtained for the same gesture types collected with an ultra-wideband radar made of a vector network analyzer with a single horn antenna and with a computer vision sensor, respectively. Based on our findings, we suggest some design implications to support future development in radar-based gesture recognition.

Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple Materials

“…The most frequently used techniques are: Convolutional Neural Networks (CNN: 33%), k-nn (18%), Long Short-Term Memory (LSTM: 14%), Support Vector Machines (SVM: 9%). A more extensive SLR [26] on 118 papers confirms these figures: 54% of the 151 algorithms come from DL, in which CNN are the most frequent, combined with LSTM or SVM. This SLR unveils a large variety of radar technology of different operating frequencies and bandwidths, antenna configurations, but also various gesture recognition techniques.…”

“…There are almost as many different approaches as there are radars with different bandwidths and frequencies. Fortunately, some surveys [35] and reviews [26] have highlighted a few trends, but these are not easily transposable to other contexts of use.…”

“…The Google Soli chip [41] is embedded in a smartphone for recognizing various classes of gestures. There are several radar-based hand tracking systems such as Ether-Pose [42], CW-Radar [43], the Magic Carpet [44] and its Gesture-Sensing Radars Project, GestureVLAD [10], Ingenious [15], Forte [45] RadarSense [26], RadarGesture [13]. Radar-based gesture interaction has been pioneered by the Magic Carpet [44], a Doppler radar used for sensing coarse body gestures by signal processing.…”

“…In view of all this work, we can apply a DL algorithm, but we are keen to put a template-based [50] recognizer [51] to the test because it allows us to customize the gestures by modifying templates only, with a reasonable number of templates [12], and without re-training [52], [53]. Furthermore, we are siding with the mobile radars that are widely available on the market by taking the Walabot DIY 2 radar [3], [5], [15], [26], [45], [54], [55].…”

“…High-frequency, wide-bandwidth radars detect even the finest movements [22], such as the fingers of a hand [23], at a reasonable distance [24]. However, the wide variety of radars used [25] and the particularity of their models, methods, and tools used to recognize gestures limit reuse in another context [26]. There are almost as many different methods as there are different radars, all reporting recognition rates worthy of direct real-time interaction.…”