Towards Domain-independent Complex and Fine-grained Gesture Recognition with RFID

Abstract

Gesture recognition plays a fundamental role in emerging Human-Computer Interaction (HCI) paradigms. Recent advances in wireless sensing show promise for device-free and pervasive gesture recognition. Among them, RFID has gained much attention given its low-cost, lightweight and pervasiveness, but pioneer studies on RFID sensing still suffer two major problems when it comes to gesture recognition. The first is they are only evaluated on simple whole-body activities, rather than complex and fine-grained hand gestures. The second is they can not effectively work without retraining in new domains, i.e. new users or environments. To tackle these problems, in this paper, we propose RFree-GR, a domain-independent RFID system for complex and finegrained gesture recognition. First of all, we exploit signals from the multi-tag array to profile the sophisticated spatio-temporal changes of hand gestures. Then, we elaborate a Multimodal Convolutional Neural Network (MCNN) to aggregate information across signals and abstract complex spatio-temporal patterns. Furthermore, we introduce an adversarial model to our deep learning architecture to remove domain-specific information while retaining information relevant to gesture recognition. We extensively evaluate RFree-GR on 16 commonly used American Sign Language (ASL) words. The average accuracy for new users and environments (new setup and new position) are 89.03%, 90.21% and 88.38%, respectively, significantly outperforming existing RFID based solutions, which demonstrates the superior effectiveness and generalizability of RFree-GR. CCS Concepts: • Human-centered computing → Gestural input; Ubiquitous and mobile computing systems and tools.

RF-Eletter: A Cross-Domain English Letter Recognition System Based on RFID

“…Gesture recognition with wireless signals, such as RFID is an emerging non-touch user interface technology that has garnered much attention following its lightness, affordability, and prevalence. CAO DIAN et al [25] proposed RFree-GR , a domain-independent RFID system that utilizes a 3*4 array of tags to capture users ' gesture signals through the designed multi-modal convolutional neural network (MCNN) to aggregate information between signals, abstract complex spatio-temporal patterns, and facilitate complex and fine-grained gesture recognition. Despite an average accuracy of 90% for novel users and environments, the system failed to recognize dynamic gestures.…”

“…A set of 2 × 3 tag arrays were utilized to study the effect of human motion on the back-scattered signal while 4 × 3 tag arrays were employed for data acquisition following [25]. Resultantly, 2 × 3 tag arrays enabled the acquisition of complex and fine-grained gesture data.…”

“…In Figure 3a, the RSS in human gesture perception demonstrated a less obvious trend of change. Thus, a phase was chosen in this experiment to perform the corresponding experimental analysis of human gesture recognition [25]. Different users ' human gesture recognition capabilities were also studied.…”

“…A series of comparative experiments were constructed to verify the gesture recognition algorithm effectiveness. Several advanced gesture recognition systems were compared based on wireless signals, such as RFree-GR [25] and RFnet [17] (see Figure 20). The average accuracy rates of RFree-GR for novel users and environments were 89.03% and 90.21%, respectively (lower than the two preceding counterparts).…”

Exploiting Serialized Fine-Grained Action Recognition Using WiFi Sensing

“…Hand gestures are an important communication channel for humans [1] and provide a natural way to support humancomputer interactions [2]. Recent years have witnessed increased interest in exploiting wireless signals such as WiFi [3][4][5][6][7] or RFID [8] for gesture recognition. Compared to other technologies, there are several advantages to wireless signals.…”