Accurate Continuous Prediction of 14 Degrees of Freedom of the Hand from Myoelectrical Signals through Convolutive Deep Learning

Sîmpetru, Raul; Oßwald, Marius; Braun, Dominik I.; Oliveira, Daniela Souza de; Cakici, Andre L.; Vecchio, Alessandro Del

doi:10.1109/embc48229.2022.9870937

Cited by 11 publications

(26 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have then implemented the model in a real-time simulated scenario, by streaming the high-density sEMG in the same bins as processed in offline experiments. As in our previous conference paper [31] we found very similar results with this improved version of the neural network. We were able to consistently predict hand movements in simulated online settings every 31.25 seconds resulting in 32 predictions per seconds.…”

Section: Three-dimensional Hand Estimationsupporting

confidence: 88%

“…Similar to our prior conference study [37, accepted for publication], we collected sEMG data from 320 electrodes and recorded simultaneously the hand movements using 5 cameras. We recorded 13 subjects out of which 11 were male and 2 female.…”

Section: Methodsmentioning

confidence: 99%

“…In this paper, we use an architecture that is similar to the one we used in our previous conference paper [31]. We have already tested that architecture in a simulated real-time environment and found it to have satisfactorily inference latency.…”

Section: Methodsmentioning

confidence: 99%

“…There have been numerous relevant attempts trying to fill this specific need [6]- [37] by exploiting the EMG signals to reconstruct the movement intent, yet all have some limitations that hinder the access to the large number of degrees of freedom from the human hand. We will delve into the shortcomings of these works later in the discussion section, but for now, here is a brief overview of the limitations of current sEMG decoding systems: 1) errors that are too big for some or all of the fingers [13]- [16], [18], [20]- [23], [25], [26], [38], 2) achieving less than 14 degrees of freedom (DoF) that the hand needs for executing flexion and extension movements [14], [15], [19], [21]- [26], [28]- [30], [38], and 3) or not attempting unseen combinations of different movements [13]- [16], [18]- [26], [28], [29], [31], [38].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sîmpetru

Arkudas

Braun

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Theories on the neural control of movement are largely based on movement-sensing devices that capture the dynamics of predefined anatomical landmarks. Neuromuscular interfaces, such as surface electromyography (sEMG), can in theory surpass the limitations imposed by motion-based technologies by sensing the motor commands transmitted by the final pathway of movement, the motor units. The recording of motor unit activity may allow the prediction of the kinetics and kinematics continuously in time and space, without being limited to several biological and physical boundaries that digital cameras or inertial sensors suffer. However, current sEMG decoding algorithms can only predict few degrees of freedom (<3). By combining markerless machine vision and high-density sEMG electrodes, we aimed to test the hypothesis that a physiologically inspired deep neural network can reconstruct the movement of the human hand as precise as digital cameras and with the additional benefit of predicting the underlying forces (i.e., grasping a cup of coffee). We demonstrate that our deep learning model can continuously predict all degrees of freedom of the hand with negligible errors during natural motion tasks from 320 sEMG sensors placed only on the extrinsic hand muscles. Our deep learning model was able to display the 3D hand kinematics and the full force range of the hand digits during isometric contractions. The current results demonstrate that deep learning applied to EMG signals gives access to an unprecedented representation of the final neural code of movement.

show abstract

Section: Three-dimensional Hand Estimationsupporting

confidence: 88%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sîmpetru

Arkudas

Braun

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Here, we demonstrate with real-time simultaneous and proportional control, that this problem is not caused by intrinsic problems associated with the EMG (i.e., that there is insufficient information contained in the signal itself) but rather by inadequate processing algorithms that are used to process the signal which is in a relatively large frequency range (20-500 Hz). In our prior conference study we investigated the offline movement decoding solutions with simulated real-time applications [28], and produced a neural network that was capable of accurate prediction on unseen data [28], [29]. That neural network is now adapted for real-time usage in this work (Fig.…”

Section: Introductionmentioning

confidence: 99%

Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography

Sîmpetru¹,

März²,

Vecchio³

2023

Preprint

View full text Add to dashboard Cite

<p>Surface electromyography (sEMG) is a non-invasive technique that measures the electrical activity generated by the muscles using sensors placed on the skin. It has been widely used in the field of prosthetics and other assistive systems because of the physiological connection between muscle electrical activity and movement dynamics. However, most existing sEMG-based decoding algorithms show a limited number of detectable degrees of freedom that can be proportionally and simultaneously controlled in real-time, which limits the use of EMG in a wide range of applications, including prosthetics and other consumer-level applications (e.g., human/machine interfacing). In this work, we surpass the current state of the art by developing a new deep learning method that can decode and map the electrophysiological activity of the forearm muscles into proportional and simultaneous control of > 20 degrees of freedom of the human hand with real-time resolution and with latency within the neuromuscular delays (< 50 ms). We recorded the kinematics of the human hand during grasping, pinching, individual digit movements and three gestures at slow (0.5 Hz) and fast (0.75 Hz) movement speeds in healthy participants.</p> <p>We demonstrate that our neural network can predict the kinematics of the hand in real-time at a constant 32 predictions per second. To achieve this, we employed transfer learning and created a prediction smoothing algorithm for the output of the neural network that reconstructed the full geometry of the hand in three-dimensional Cartesian space in real-time. Our results demonstrate that high-density EMG signals from the forearm muscles contain almost all the information that is needed to predict the kinematics of the human hand. The proposed method has the capability of predicting the full kinematics of the human hand in an unprecedented way and with real-time resolution with immediate translational impact in subjects with motor impairments. </p>

show abstract

Influence of spatio-temporal filtering on hand kinematics estimation from high-density EMG signals ^*

Sîmpetru,

Cnejevici,

Farina

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

Objective: Surface electromyography (sEMG) is a non-invasive technique that records the electrical signals generated by muscles through electrodes placed on the skin. sEMG is the state-of-the-art method used to control active upper limb prostheses because of the association between its amplitude and the neural drive sent from the spinal cord to muscles. However, accurately estimating the kinematics of a freely moving human hand using sEMG from extrinsic hand muscles remains a challenge. Deep learning has been recently successfully applied to this problem by mapping raw sEMG signals into kinematics. Nonetheless, the optimal number of EMG signals and the type of pre-processing that would maximize performance have not been investigated yet. Approach: Here, we analyze the impact of these factors on the accuracy in kinematics estimates. For this purpose, we processed monopolar sEMG signals that were originally recorded from 320 electrodes over the forearm muscles of 13 subjects. We used a previously published deep learning method that can map the kinematics of the human hand with real-time resolution. Main results: While myocontrol algorithms essentially use the temporal envelope of the EMG signal as the only EMG feature, we show that our approach requires the full bandwidth of the signal in the temporal domain for accurate estimates. Spatial filtering however, had a smaller impact and low-order spatial filters may be suitable. Moreover, reducing the number of channels by ablation resulted in large performance losses.The highest accuracy was reached with the highest number of available sensors (n=320). Importantly and unexpected, our results suggest that increasing the number of channels above those used in this study may further enhance the accuracy in predicting the kinematics of the human hand. Significance: We conclude that full bandwidth high-density EMG systems of hundreds of electrodes are needed for accurate kinematic estimates of the human hand.

show abstract

Accurate Continuous Prediction of 14 Degrees of Freedom of the Hand from Myoelectrical Signals through Convolutive Deep Learning

Cited by 11 publications

References 18 publications

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography

Influence of spatio-temporal filtering on hand kinematics estimation from high-density EMG signals ^*

Contact Info

Product

Resources

About

Accurate Continuous Prediction of 14 Degrees of Freedom of the Hand from Myoelectrical Signals through Convolutive Deep Learning

Cited by 11 publications

References 18 publications

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Proportional and Simultaneous Real-Time Control of the Full Human Hand From High-Density Electromyography

Influence of spatio-temporal filtering on hand kinematics estimation from high-density EMG signals *

Contact Info

Product

Resources

About

Influence of spatio-temporal filtering on hand kinematics estimation from high-density EMG signals ^*