Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Abstract

Theories on the neural control of movement are largely based on movement-sensing devices that capture the dynamics of predefined anatomical landmarks. Neuromuscular interfaces, such as surface electromyography (sEMG), can in theory surpass the limitations imposed by motion-based technologies by sensing the motor commands transmitted by the final pathway of movement, the motor units. The recording of motor unit activity may allow the prediction of the kinetics and kinematics continuously in time and space, without being limited to several biological and physical boundaries that digital cameras or inertial sensors suffer. However, current sEMG decoding algorithms can only predict few degrees of freedom (<3). By combining markerless machine vision and high-density sEMG electrodes, we aimed to test the hypothesis that a physiologically inspired deep neural network can reconstruct the movement of the human hand as precise as digital cameras and with the additional benefit of predicting the underlying forces (i.e., grasping a cup of coffee). We demonstrate that our deep learning model can continuously predict all degrees of freedom of the hand with negligible errors during natural motion tasks from 320 sEMG sensors placed only on the extrinsic hand muscles. Our deep learning model was able to display the 3D hand kinematics and the full force range of the hand digits during isometric contractions. The current results demonstrate that deep learning applied to EMG signals gives access to an unprecedented representation of the final neural code of movement.

Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography

“…In a previous work [29], we also studied nonlinfactorization techniques are the features learned by the neural network. found that the model learned to separate each hand movement distinctively, with clear clusters between fingers and between the flexion/extension movements for the same finger.…”

Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography

“…In short, Gaussian Noise [30, p. 3] modifies the signal to have a signal-to-noise ratio of 5, Magnitude Warping [30, p. 3] accounts for the shift in the electrode grids, and Wavelet Decomposition [30, p. 4] facilitates model generalization by reconstructing the original signal with noise. (d) Schematic overview of our adapted model from Sîmpetru et al [29] for real-time inference. Each input grid displays 8 random electrode signals from that particular grid.…”

“…The filtered version was then appended to the raw (digitally unfiltered) segment in the depth dimension, resulting in an sEMG tensor of the shape depth (raw or filtered) × number of electrodes × time in samples. We have before demonstrated in offline experiments that the best results for a deep learning architecture similar to the one used in this study was when we input raw monopolar EMG signals and a rectified, low-pass filtered version (20 Hz) of the same signals concurrently [28], [29].…”

“…This work is a real-time adaptation of our previous works [28], [29], both based on the same basic model, which we updated to the latest available machine learning frameworks for Python (PyTorch [35] 1.12.0+cu116 and PyTorch-Lightning [36] 1.7.1). For this reason, we will explain our adjustments in detail, but keep the general description of the architecture short, as a detailed explanation can be found in [29]. A graphical overview of the architecture of the model is given in Fig.…”

“…Here, we demonstrate with real-time simultaneous and proportional control, that this problem is not caused by intrinsic problems associated with the EMG (i.e., that there is insufficient information contained in the signal itself) but rather by inadequate processing algorithms that are used to process the signal which is in a relatively large frequency range (20-500 Hz). In our prior conference study we investigated the offline movement decoding solutions with simulated real-time applications [28], and produced a neural network that was capable of accurate prediction on unseen data [28], [29]. That neural network is now adapted for real-time usage in this work (Fig.…”