Gesture Spotter: A Rapid Prototyping Tool for Key Gesture Spotting in Virtual and Augmented Reality Applications

Abstract

In this paper we examine the task of key gesture spotting: accurate and timely online recognition of hand gestures. We specifically seek to address two key challenges faced by developers when integrating key gesture spotting functionality into their applications. These are: i) achieving high accuracy and zero or negative activation lag with single-time activation; and ii) avoiding the requirement for deep domain expertise in machine learning. We address the first challenge by proposing a key gesture spotting architecture consisting of a novel gesture classifier model and a novel single-time activation algorithm. This key gesture spotting architecture was evaluated on four separate hand skeleton gesture datasets, and achieved high recognition accuracy with early detection. We address the second challenge by encapsulating different data processing and augmentation strategies, as well as the proposed key gesture spotting architecture, into a graphical user interface and an application programming interface. Two user studies demonstrate that developers are able to efficiently construct custom recognizers using both the graphical user interface and the application programming interface.

HotGestures: Complementing Command Selection and Use with Delimiter-Free Gesture-Based Shortcuts in Virtual Reality

“…Molchanov et al [36] investigated early detection of gestures in unsegmented video streams. Shen et al [51] developed a toolkit for fast prototyping of recognition models for online hand gestures based on variations of CNN and LSTM architectures. Prior work has looked at multi-task prediction using neural networks for additional outputs [59,60] and a way to facilitate transfer learning [39,64,65,69].…”

Structural design of magnetostrictive sensing glove and its application for gesture recognition

“…Gesture serves a crucial function in daily life as a versatile tool for conveying information (Theodoridou et al , 2023; Duarte et al , 2021). Gesture recognition plays a very crucial role in medical rehabilitation (Chen et al , 2020; Amsterdam et al , 2021), virtual and augmented reality (Shen et al , 2022; Park et al , 2021), human–computer interaction (Villani et al , 2023; Carfì and Mastrogiovanni, 2023), drone control (Yu et al , 2022; Haratiannejadi and Selmic, 2020) and other fields. Gesture recognition using wearable devices is a common and effective recognition method, and wearable devices use a variety of sensors for gesture monitoring, common sensors are strain, electromyographic, curvature resistive and textile fabric sensors (Dong et al , 2022; Randhawa et al , 2020).…”

Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

“…W ITH the rapid advancement of Human-Computer Interaction (HCI) techniques, human hands play a vital role in performing operations such as grasping and manipulating objects [42], [58]. In many HCI applications [57] such as Virtual/Augmented Reality (VR/AR), providing an immersive experience for users relies on quickly tracking human hands and accurately estimating the corresponding hand poses [27], [40].…”