Inertia-Constrained Reinforcement Learning to Enhance Human Motor Control Modeling

Abstract

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

A Survey on Physics Informed Reinforcement Learning: Review and Open Problems

“…The physics based reward component (= evolution of population density/ mean-field state) is approximated using PINN. To better mimic natural human locomotion [68], designed reward function based on physical and experimental information: trajectory optimization rewards, and bio-inspired rewards. In a similar task of imitation of human motion but from motion clip, [20] proposes a physics-based controller using DRL.…”

“…A well-defined reward function is crucial for successful reinforcement learning, PIRL approaches also seek to incorporate physical constraints into the design for safe learning and more efficient reward functions. For example, in [68] the designed reward incorporates IMU sensor data, imbibing inertial constraints, while in [75] the physics informed reward is designed to satisfy explicit operational targets. To ensure safe exploration during training and deployment, works such as [133,141] learn a data-driven barrier certificate based on physical property-based losses and a set of unsafe state vectors.…”

Neural correlates of fine motor grasping skills: Longitudinal insights into motor cortex activation using fNIRS

“…However, the field is not without its limitations. Previous research has predominantly adopted a transient, cross‐sectional approach, often sidelining the significance of real‐world tasks and their implications on motor learning (Krakauer et al., 2019; Korivand et al., 2023). Our study seeks to mitigate these shortcomings through a longitudinal exploration, aiming to unravel the temporal dynamics of cortical activation across various stages of motor learning, with a pronounced emphasis on real‐world tasks.…”