LE-HGR: A Lightweight and Efficient RGB-Based Online Gesture Recognition Network for Embedded AR Devices

Abstract

Figure 1: LE-HGR Framework. The overall framework of the proposed gesture recognition algorithm. The framework contains four sub-modules: Hand candidate detector, multi-task CNN, hand trace mapping, and trace sequence network. ABSTRACTOnline hand gesture recognition (HGR) techniques are essential in augmented reality (AR) applications for enabling natural humanto-computer interaction and communication. In recent years, the consumer market for low-cost AR devices has been rapidly growing, while the technology maturity in this domain is still limited. Those devices are typical of low prices, limited memory, and resourceconstrained computational units, which makes online HGR a challenging problem. To tackle this problem, we propose a lightweight and computationally efficient HGR framework, namely LE-HGR, to enable real-time gesture recognition on embedded devices with low computing power. We also show that the proposed method is of high accuracy and robustness, which is able to reach high-end performance in a variety of complicated interaction environments.To achieve our goal, we first propose a cascaded multi-task convolutional neural network (CNN) to simultaneously predict probabilities of hand detection and regress hand keypoint locations online. We show that, with the proposed cascaded architecture design, false-positive estimates can be largely eliminated. Additionally, an associated mapping approach is introduced to track the hand trace via the predicted locations, which addresses the interference of multi-handedness. Subsequently, we propose a trace sequence neural network (TraceSeqNN) to recognize the hand gesture by exploiting the motion features of the tracked trace. Finally, we provide a variety of experimental results to show that the proposed framework is able to achieve state-of-the-art accuracy with significantly reduced computational cost, which are the key properties for enabling real-time applications in low-cost commercial devices such as mobile devices and AR/VR headsets. * These authors contributed equally to this work.

Gesture Spotter: A Rapid Prototyping Tool for Key Gesture Spotting in Virtual and Augmented Reality Applications

“…Chalasani et al [7] generated augmented data by using a green screen to capture hand gestures and then overlaid these on different backgrounds. Xie et al [49] performed simultaneous detection of hands and tracking of keypoints to support gesture recognition while Wang et al [47] examined online gesture recognition for interaction with a vehicle HUD. However, all these prior studies use image data.…”

Fine-grained Hand Gesture Recognition in Multi-viewpoint Hand Hygiene

“…Specifically, in [4], [11], [15]- [19], the authors use information from depth sensors as input to give out hand gesture predictions. In [4], [8], [20]- [25], the skeleton information is used as input. Recently, Leap Motion has also been considered as an essential sensor for hand gesture recognition [18], [26].…”

Security and blockchain convergence with Internet of Multimedia Things: Current trends, research challenges and future directions

“…AI has a crucial role in processing the voluminous data gathered from IoMT devices. Embedded AI algorithms will empower customized AR experiences of our daily lives ranging from personalized dietary plans to artistic virtual overlays [174]. This convergence will revolutionize the experience of using Smart AR Glasses in near future.…”