No Interface, No Problem: Gesture Recognition on Physical Objects Using Radar Sensing

Abstract

Physical objects are usually not designed with interaction capabilities to control digital content. Nevertheless, they provide an untapped source for interactions since every object could be used to control our digital lives. We call this the missing interface problem: Instead of embedding computational capacity into objects, we can simply detect users’ gestures on them. However, gesture detection on such unmodified objects has to date been limited in the spatial resolution and detection fidelity. To address this gap, we conducted research on micro-gesture detection on physical objects based on Google Soli’s radar sensor. We introduced two novel deep learning architectures to process range Doppler images, namely a three-dimensional convolutional neural network (Conv3D) and a spectrogram-based ConvNet. The results show that our architectures enable robust on-object gesture detection, achieving an accuracy of approximately 94% for a five-gesture set, surpassing previous state-of-the-art performance results by up to 39%. We also showed that the decibel (dB) Doppler range setting has a significant effect on system performance, as accuracy can vary up to 20% across the dB range. As a result, we provide guidelines on how to best calibrate the radar sensor.

Assessing the Acceptance of a Mid-Air Gesture Syntax for Smart Space Interaction: An Empirical Study

“…This participant count strikes a balance, allowing us to gather insights into key perception aspects, and to identify significant design hurdles that could impede user experience. Other related studies with similar numbers of users include [13,22]. Previous research [42] suggests that 10 users can uncover 80% or more of the usability issues, while, for comparative studies like ours, which rely on metrics, "group sizes of between eight and 25 participants typically provide valid results, with ten to twelve being a good baseline" [43].…”

“…No validation with users was developed in these studies. In Attygalle et al [13], the authors focus on improving single-gesture recognition by training a 3D convolutional neural network and testing its performance with 10 users. Ruiz et al [14] carried out a guessability study that elicits end-user gestures to invoke commands on a smartphone device (the actions to be performed are concern the phone's resources and behaviours).…”

Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple Materials

“…Radars are today used in both stationary and mobile contexts of use. Prior works on radar-based gesture recognition [14], [18], [38], [39] mostly relied on a fixed, custom radar and on advanced ML/DL algorithms to cope with the complexity of radar signals. More recent works support dynamic, real-time recognition in mobile contexts of use [15], [18], [40].…”

“…Prior works on radar-based gesture recognition [14], [18], [38], [39] mostly relied on a fixed, custom radar and on advanced ML/DL algorithms to cope with the complexity of radar signals. More recent works support dynamic, real-time recognition in mobile contexts of use [15], [18], [40]. The Google Soli chip [41] is embedded in a smartphone for recognizing various classes of gestures.…”

“…On one hand, radar-based gesture recognition systems have demonstrated their effectiveness [10], reliability, and robustness [11] despite being influenced by environmental conditions [12]. Most studies [12]- [18] have reported their perfor-mance in terms of high recognition accuracy rates, making dynamic [16], [17], real-time [19]- [21] interaction feasible. High-frequency, wide-bandwidth radars detect even the finest movements [22], such as the fingers of a hand [23], at a reasonable distance [24].…”