An sEMG-Controlled 3D Game for Rehabilitation Therapies: Real-Time Time Hand Gesture Recognition Using Deep Learning Techniques

Abstract

In recent years the advances in Artificial Intelligence (AI) have been seen to play an important role in human well-being, in particular enabling novel forms of human-computer interaction for people with a disability. In this paper, we propose a sEMG-controlled 3D game that leverages a deep learning-based architecture for real-time gesture recognition. The 3D game experience developed in the study is focused on rehabilitation exercises, allowing individuals with certain disabilities to use low-cost sEMG sensors to control the game experience. For this purpose, we acquired a novel dataset of seven gestures using the Myo armband device, which we utilized to train the proposed deep learning model. The signals captured were used as an input of a Conv-GRU architecture to classify the gestures. Further, we ran a live system with the participation of different individuals and analyzed the neural network’s classification for hand gestures. Finally, we also evaluated our system, testing it for 20 rounds with new participants and analyzed its results in a user study.

Effects of EMG-Controlled Video Games on the Upper Limb Functionality in Patients with Multiple Sclerosis: A Feasibility Study and Development Description

“…MYO can communicate on real-time hand gesture recognition using deep learning techniques with PCs via Bluetooth, virtual environments, or other objectives such as prosthesis [ 13 ], a steering assistance interface [ 14 ], an augmented reality dance game designed to improve rehabilitation therapies in upper limb amputees while the hand gestures are analyzed using EMG data collected by MYO [ 15 ], or an EMG intention detection system based on the MYO armband to control robotic hand orthosis [ 16 ]. It has been recognized as a noninvasive, more user-friendly, and time-saving device compared with conventional electrodes [ 17 ].…”

Hand Gesture Recognition Using Single Patchable Six-Axis Inertial Measurement Unit via Recurrent Neural Networks

“…Second, the hand gestures were classified as smoking, eating, and drinking gestures, reporting a smoking accuracy of 91.38%. Nasri et al [ 41 ] used hand gesture recognition for an HCI-based system. In their system, a GRU-based algorithm recognized seven hand gestures to control a 3D game, achieving an accuracy of 82.15% with a new user.…”

A Hand Motor Skills Rehabilitation for the Injured Implemented on a Social Robot

“…These wearable electrodes enable the professional in charge to analyze the muscular activities and muscular strength of a patient during each exercise [18,19]. In some rehabilitation methods, with aim of having an interactive system and motivating patients with visual and audio feedback, during the process, sEMG analysis is incorporated into a 3D game [20,21], virtual reality (VR) [22], and augmented reality (AR) [23]. In addition, there are algorithms that take sEMG signals as the input in order to compute [24] different low-level move-ments of the muscles.…”