Electromyography Based Decoding of Dexterous, In-Hand Manipulation Motions With Temporal Multichannel Vision Transformers

Abstract

Electromyography (EMG) signals have been used in designing muscle-machine interfaces (MuMIs) for various applications, ranging from entertainment (EMG controlled games) to human assistance and human augmentation (EMG controlled prostheses and exoskeletons). For this, classical machine learning methods such as Random Forest (RF) models have been used to decode EMG signals. However, these methods depend on several stages of signal pre-processing and extraction of handcrafted features so as to obtain the desired output. In this work, we propose EMG based frameworks for the decoding of object motions in the execution of dexterous, inhand manipulation tasks using raw EMG signals input and two novel deep learning (DL) techniques called Temporal Multi-Channel Transformers and Vision Transformers. The results obtained are compared, in terms of accuracy and speed of decoding the motion, with RF-based models and Convolutional Neural Networks as a benchmark. The models are trained for 11 subjects in a motion-object specific and motion-object generic way, using the 10-fold crossvalidation procedure. This study shows that the performance of MuMIs can be improved by employing DL-based models with raw myoelectric activations instead of developing DL or classic machine learning models with handcrafted features.

Movement Recognition via Channel-Activation-Wise sEMG Attention

“…In 2022, Transformer has seen extensions to sEMG recognition [31,39]. Specifically, in work [31] a novel convolutional vision transformer (CviT) with stacking ensemble learning was proposed for the fusion of sequential and spatial features of sEMG signals with parallel training.…”

“…In addition, the successful application of Transformer in sEMGbased movements classification provides a significant reference for the application of Transformer in other biological signals [31]. In [39], the transformer architecture was leveraged for decoding object motions in dexterous in-hand manipulation tasks using raw EMG signals input. Their new architecture Temporal Multi-Channel Transformers and Vision Transformers were shown to outperform RF-based models and CNNs in terms of accuracy and speed of decoding the motion.…”

Challenges and Trends of Machine Learning in the Myoelectric Control System for Upper Limb Exoskeletons and Exosuits

“…The abovementioned articles utilized the statistical features of EMG data as input for the classification-based myoelectric control systems. However, the raw EMG signal can also be used as input for the classification-based myoelectric control system, as demonstrated by [41], which successfully implemented a vision transformer model to classify two datasets using raw multichannel EMG data. The transformer model is commonly used in natural language processing, but the encoder-decoder network can be applied to determine the underlying characteristics of the input data without manual feature extraction or signal pre-processing.…”