Multi-Category Gesture Recognition Modeling Based on sEMG and IMU Signals

Abstract

Gesture recognition based on wearable devices is one of the vital components of human–computer interaction systems. Compared with skeleton-based recognition in computer vision, gesture recognition using wearable sensors has attracted wide attention for its robustness and convenience. Recently, many studies have proposed deep learning methods based on surface electromyography (sEMG) signals for gesture classification; however, most of the existing datasets are built for surface EMG signals, and there is a lack of datasets for multi-category gestures. Due to model limitations and inadequate classification data, the recognition accuracy of these methods cannot satisfy multi-gesture interaction scenarios. In this paper, a multi-category dataset containing 20 gestures is recorded with the help of a wearable device that can acquire surface electromyographic and inertial (IMU) signals. Various two-stream deep learning models are established and improved further. The basic convolutional neural network (CNN), recurrent neural network (RNN), and Transformer models are experimented on with our dataset as the classifier. The CNN and the RNN models’ test accuracy is over 95%; however, the Transformer model has a lower test accuracy of 71.68%. After further improvements, the CNN model is introduced into the residual network and augmented to the CNN-Res model, achieving 98.24% accuracy; moreover, it has the shortest training and testing time. Then, after combining the RNN model and the CNN-Res model, the long short term memory (LSTM)-Res model and gate recurrent unit (GRU)-Res model achieve the highest classification accuracy of 99.67% and 99.49%, respectively. Finally, the fusion of the Transformer model and the CNN model enables the Transformer-CNN model to be constructed. Such improvement dramatically boosts the performance of the Transformer module, increasing the recognition accuracy from 71.86% to 98.96%.

High-Performance Surface Electromyography Armband Design for Gesture Recognition

“…In recent years, researchers have used various classifiers to classify sEMG, including k-nearest neighbor (KNN), linear discriminant analysis (LDA), and support vector machines (SVMs), etc. [ 18 , 19 ]. Deep learning has been widely applied in computer vision.…”

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

“…The majority of these approaches combine traditional feature extraction with DNNs, with subject-specific assessments [10,30,49,50]. Notably, state-of-the-art HGR methods using end-toend sEMG and ACC DL models, such as NIDA-TCN [31] and two-stream LSTM-Res [35], lack fair comparative evaluations on public databases like Ninapro. Hence, we replicated these models for comparative analysis with our proposed NIMFT on the Ninapro DB2.…”

“…However, most of these studies often required extensive feature extraction, lacked intersubject experiments, and did not evaluate the transferability of the models. Notably, existing end-to-end sEMG-ACC fusion deep learning models, like those in [31,35], have not performed inter-subject tests on public datasets for benchmarking against established model. The absence of inter-subject evaluation and transferability assessment reduces the real-world applicability of these models.…”

Supporting Tremor Rehabilitation Using Optical See-Through Augmented Reality Technology

“…They also revealed the limitations of virtual reality technology in rehabilitation, including a lack of supportive infrastructure, expensive equipment, and inadequate communication infrastructure for rural tele-assistance. Jiang et al [22]. presented a multi-category gesture recognition model that uses signals from both surface electromyography and inertial measurement units.…”