Artificial Intelligence for Prosthetics: Challenge Solutions

Abstract

In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector. Top participants were invited to describe their algorithms. In this work, we describe the challenge and present thirteen solutions that used deep reinforcement learning approaches. Many solutions use similar relaxations and heuristics, such as reward shaping, frame skipping, discretization of the action space, symmetry, and policy blending. However, each team implemented different modifications of the known algorithms by, for example, dividing the task into subtasks, learning low-level control, or by incorporating expert knowledge and using imitation learning.

Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation

“…Various RL techniques have been effectively used since the first competition [124,125], including frame skipping, discretization of the action space, and reward shaping. These are practical techniques that constrain the problem in certain ways to encourage an agent to search successful regions faster in the initial stages of training.…”

OpenSim Moco: Musculoskeletal optimal control

“…Moco attempts to produce well-scaled optimization problems to facilitate rapid convergence (e.g., using normalized tendon force as a state variable), but providing automated problem scaling ( [20], section 4.8) could further improve convergence. Lastly, the direct collocation method itself is not well-suited to simulations that are interactive (e.g., users perturbing the model) or include randomness (e.g., uneven terrain); such problems are better suited to single shooting [12] or reinforcement learning approaches [77].…”

Deep Reinforcement Learning for Physics-Based Musculoskeletal Simulations of Healthy Subjects and Transfemoral Prostheses’ Users During Normal Walking

“…In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were asked to build a controller for a transtibial amputee model with the goal of moving it forward, and were encouraged to use DRL. Kidzinski et al, [8] use a musculoskeletal model of 19 muscles with one leg having the below-knee leg replaced by a prosthesis. Results have shown that DRL can find solutions in which the agent learns a policy to efficiently move forward.…”

“…A state-of-the-art DRL algorithm (PPO) is implemented and a DRL algorithm (PPO with imitation learning) is proposed and implemented on a musculoskeletal model of a healthy subject in the open-source simulation software OpenSim [7]. This paper uses the model presented in [8], which consists of two healthy legs including 18 healthy muscles (9 per leg) to control 10 degrees of freedom. The DRL algorithms (PPO and PPO with imitation learning) are validated on a public data-set [9].…”

“…The muscles are: 11 healthy muscles in the sound leg, 4 muscles at the hip joint of the amputated leg, and 4 musclelike actuators in the transfemoral prosthesis, i.e., a generic prosthetic device with two agonist/antagonist muscle-like actuators at the knee joint and two at the ankle joint, which are lost following the transfemoral amputation. Previous research has shown that it is possible to build realistic human models with several deficits [2], transtibial (below-knee) amputation [10], [8], and transfemoral amputation [11] in OpenSim. However, there is not yet a transfemoral amputees model with musclelike actuators on a prosthetic leg, specifically designed for implementing DRL, as proposed in this paper.…”