Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity

Abstract

We address the question whether the assumptions of signal-dependent and constant motor noise in a full skeletal model of the human upper extremity, together with the objective of movement time minimization, can predict reaching movements. We learn a control policy using a motor babbling approach based on reinforcement learning, using aimed movements of the tip of the right index finger towards randomly placed 3D targets of varying size. The reward signal is the negative time to reach the target, implying movement time minimization. Our biomechanical model of the upper extremity uses the skeletal structure of the Upper Extremity Dynamic Model, including thorax, right shoulder, arm, and hand. The model has 7 actuated degrees of freedom, including shoulder rotation, elevation and elevation plane, elbow flexion, forearm rotation, and wrist flexion and deviation. To deal with the curse of dimensionality, we use a simplified second-order muscle model acting at each joint instead of individual muscles. We address the lack of gradient provided by the simple reward function through an adaptive learning curriculum. Our results demonstrate that the assumptions of signal-dependent and constant motor noise, together with the objective of movement time minimization, are sufficient for a state-of-the-art skeletal model of the human upper extremity to reproduce complex phenomena of human movement such as Fitts' Law and the 2 /3 Power Law. This result supports the idea that the control of the complex human biomechanical system is plausible to be determined by a set of simple assumptions and can be easily learned.

Optimal Feedback Control for Modeling Human-Computer Interaction

“…They do not make particular a priori assumptions about the structure of the cost function. Instead, they use an inverse optimal control approach to fit a generic function with a large number of parameters (36) to a dataset of mouse pointer movements. While Ziebart et al [145] focus on the application of inverse optimal control to pointing target prediction, in this paper we investigate the ability of optimal (feedback) control models to model movement of the mouse pointer more quantitatively.…”

“…Cheema et al [24] have applied recent RL methods to predict fatigue during mid-air movements, using a torque-actuated linked-segment model of the upper limb. Building on this work, it has recently been shown that RL applied to a more realistic upper-limb model allows to synthesize human arm movements that follow both Fitts' Law and the 2 /3 Power Law and can predict human behavior in mid-air pointing and path following tasks [36]. Moreover, an extension to mid-air keyboard typing has been proposed [55].…”

“…Traditionally, the field of Human-Computer Interaction (HCI) has concentrated on models such as Fitts' Law [37,38], predicting summary statistics of the movement such as movement time. Recently, more attention has been paid to modeling the underlying process by which the end-effector is controlled, predicting not only movement time, but end-effector position, velocity, and acceleration sequences, as well as applied forces (e.g., [32,36,83,91]).…”

MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies

“…Alternatively, physics engines such as PyBullet [15], MuJoCo [16] and Dart [17] are relatively more efficient and support contact interactions but lack adequate support for muscle modeling (PyBullet and Dart) or lack functionally validated musculoskeletal models. Even though attempts have been made in adding physiological models in MuJoCo [18]- [20], they were limited in their validation.…”