Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling

Wannawas, Nat; Subramanian, Mahendran; Faisal, A. Aldo

doi:10.1109/ner49283.2021.9441354

Cited by 13 publications

(8 citation statements)

References 26 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A feasible alternative to the approach adopted in this paper, which models the kinematics and dynamics of the skeletal muscle model, analyzes its system characteristics, and then designs a controller, is to use software already available for skeletal muscle systems to train an efficient controller directly through thousands of unrestricted interactions with the musculoskeletal system. A good example of adopting this method is when Wannawas et al [ 36 ] used neuromechanics-based deep reinforcement learning to control FES during cycling. They first developed a neuromechanical model of cycling based on an open software named “OpenSim”, and then optimized the control strategy via allowing interactions between the musculoskeletal model and the mechanical properties of a bicycle.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

Neuromechanics-Based Neural Feedback Controller for Planar Arm Reaching Movements

Zhao

Zhang

et al. 2023

Bioengineering

View full text Add to dashboard Cite

Based on the principles of neuromechanics, human arm movements result from the dynamic interaction between the nervous, muscular, and skeletal systems. To develop an effective neural feedback controller for neuro-rehabilitation training, it is important to consider both the effects of muscles and skeletons. In this study, we designed a neuromechanics-based neural feedback controller for arm reaching movements. To achieve this, we first constructed a musculoskeletal arm model based on the actual biomechanical structure of the human arm. Subsequently, a hybrid neural feedback controller was developed that mimics the multifunctional areas of the human arm. The performance of this controller was then validated through numerical simulation experiments. The simulation results demonstrated a bell-shaped movement trajectory, consistent with the natural motion of human arm movements. Furthermore, the experiment testing the tracking ability of the controller revealed real-time errors within one millimeter, with the tensile force generated by the controller’s muscles being stable and maintained at a low value, thereby avoiding the issue of muscle strain that can occur due to excessive excitation during the neurorehabilitation process.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

Neuromechanics-Based Neural Feedback Controller for Planar Arm Reaching Movements

Zhao

Zhang

et al. 2023

Bioengineering

View full text Add to dashboard Cite

show abstract

“…To overcome this drawback, a better approach would be to use mathematical models to simulate a priori different combinations of parameters, as well as the agent’s exploration and exploitation characteristics to learn the policy. In [ 13 ], researchers suggested the RL controller be trained to learn a defined stimulation strategy to cycle at the desired cadences in simulation before evaluating the performance in the real world. They developed a deep RL algorithm to control the stimulation of different leg muscles during FES-cycling sessions and demonstrated the ability of the agent to learn from its iterations with the system.…”

Section: Discussionmentioning

confidence: 99%

“…A recent study introduced a method using RL for real-time optimization of stimulation patterns while monitoring the average cadence error during simulated FES-cycling sessions [ 13 ]. The performance outcome obtained with a musculoskeletal model simulation was compared to other controllers, such as PID and Fuzzy Logic.…”

Section: Introductionmentioning

confidence: 99%

A Novel Functional Electrical Stimulation-Induced Cycling Controller Using Reinforcement Learning to Optimize Online Muscle Activation Pattern

Coelho-Magalhães

Coste

Martins

2022

Sensors

View full text Add to dashboard Cite

This study introduces a novel controller based on a Reinforcement Learning (RL) algorithm for real-time adaptation of the stimulation pattern during FES-cycling. Core to our approach is the introduction of an RL agent that interacts with the cycling environment and learns through trial and error how to modulate the electrical charge applied to the stimulated muscle groups according to a predefined policy and while tracking a reference cadence. Instead of a static stimulation pattern to be modified by a control law, we hypothesized that a non-stationary baseline set of parameters would better adjust the amount of injected electrical charge to the time-varying characteristics of the musculature. Overground FES-assisted cycling sessions were performed by a subject with spinal cord injury (SCI AIS-A, T8). For tracking a predefined pedaling cadence, two closed-loop control laws were simultaneously used to modulate the pulse intensity of the stimulation channels responsible for evoking the muscle contractions. First, a Proportional-Integral (PI) controller was used to control the current amplitude of the stimulation channels over an initial parameter setting with predefined pulse amplitude, width and fixed frequency parameters. In parallel, an RL algorithm with a decayed-epsilon-greedy strategy was implemented to randomly explore nine different variations of pulse amplitude and width parameters over the same stimulation setting, aiming to adjust the injected electrical charge according to a predefined policy. The performance of this global control strategy was evaluated in two different RL settings and explored in two different cycling scenarios. The participant was able to pedal overground for distances over 3.5 km, and the results evidenced the RL agent learned to modify the stimulation pattern according to the predefined policy and was simultaneously able to track a predefined pedaling cadence. Despite the simplicity of our approach and the existence of more sophisticated RL algorithms, our method can be used to reduce the time needed to define stimulation patterns. Our results suggest interesting research possibilities to be explored in the future to improve cycling performance since more efficient stimulation cost dynamics can be explored and implemented for the agent to learn.

show abstract

“…Currently, successful applications of machine learning‐based methods are available, especially in the case of neural network‐based learning (Wannawas et al, 2021). Another remarkable example is the work of Xia et al; the authors introduce a recurrent and convolutional neural network‐based method that is able to derive the spatial position coordinates of a moving hand from the EMG records of the same arm (Xia et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Prediction of leg muscle activities from arm muscle activities in arm and leg cycling

Radeleczki

Mravcsik

Bozheim

et al. 2022

The Anatomical Record

View full text Add to dashboard Cite

Functional electrical stimulation (FES) driven leg cycling is usually controlled by previously established stimulation patterns. We investigated the potential utilization of a particular computational method for controlling electrical stimulation of lower limb muscles by real-time electromyography (EMG) signals of arm muscles during hybrid arm and leg cycling. In hybrid arm and leg cycling, arm cranking is performed voluntarily, while leg cycling is driven by FES. In this study, we investigate arm and leg cycling movements of able-bodied persons when both arm and leg cycling is performed voluntarily without FES. We present a neural network-based model in which the input of the neural network is given by a time series of upper limb muscle activities (EMG), and the output provides potential lower limb muscle activities. The particular neural network was a nonlinear autoregressive exogen (NARX) neural network. The measured EMG signals of the lower limb muscles were compared to the signals that were predicted by the neural network. The neural network was trained with data recorded from four participants. Our preliminary results show notable differences between the predicted and the experimentally measured lower limb muscle activities. The prediction was good only for 60% of the movement time. We conclude that-while including arm cycling in the movementsimpler control modalities or further consideration of applying machinelearning techniques has to be taken into account to improve voluntary upper limb-controlled FES assisted leg cycling.

show abstract

Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling

Cited by 13 publications

References 26 publications

Neuromechanics-Based Neural Feedback Controller for Planar Arm Reaching Movements

Neuromechanics-Based Neural Feedback Controller for Planar Arm Reaching Movements

A Novel Functional Electrical Stimulation-Induced Cycling Controller Using Reinforcement Learning to Optimize Online Muscle Activation Pattern

Prediction of leg muscle activities from arm muscle activities in arm and leg cycling

Contact Info

Product

Resources

About