Learning to Run Challenge: Synthesizing Physiologically Accurate Motion Using Deep Reinforcement Learning

Abstract

Synthesizing physiologically-accurate human movement in a variety of conditions can help practitioners plan surgeries, design experiments, or prototype assistive devices in simulated environments, reducing time and costs and improving treatment outcomes. Because of the large and complex solution spaces of biomechanical models, current methods are constrained to specific movements and models, requiring careful design of a controller and hindering many possible applications. We sought to discover if modern optimization methods efficiently explore these complex spaces. To do this, we posed the problem as a competition in which participants were tasked with developing a controller to enable a physiologically-based human model to navigate a complex obstacle course as quickly as possible, without using any experimental data. They were provided with a human musculoskeletal model and a physics-based simulation environment. In this paper, we discuss the design of the competition, technical difficulties, results, and analysis of the top controllers. The challenge proved that deep reinforcement learning techniques, despite their high computational cost, can be successfully employed as an optimization method for synthesizing physiologically feasible motion in high-dimensional biomechanical systems. * These authors contributed equally to this work

Predicting gait adaptations due to ankle plantarflexor muscle weakness and contracture using physics-based musculoskeletal simulations

“…Reflex-based controllers, like the one used here, were useful to ensure that the optimization problem was tractable, but a controller like this cannot predict changes in the underlying control structure. Recent advances in the field of reinforcement learning may be able to generate robust simulations with control patterns that do not have a specified and simplified control structure [54,55].…”

Artificial Intelligence for Prosthetics: Challenge Solutions

“…osim-rl [23] consumes around 0.22s for simulating one step where the actor side is the bottleneck and the throughput (time steps per second) as well as convergence speed increases significantly as we add more actors (see Fig. 11a).…”

The AI Driving Olympics at NeurIPS 2018

“…Robotics is often cited as a potential target application in ML literature (e.g., [6,12] and many others). The vast majority of these works evaluate their results in simulation [14] and on very simple (usually grid-like) environments [26]. However, simulations are by design based on what we already know and lack some of the DARPA [4] KITTI [10] Robocup [15] RL comp.…”

“…However, simulations are by design based on what we already know and lack some of the DARPA [4] KITTI [10] Robocup [15] RL comp. [14] HuroCup [1] AI-DO Table 1: Characteristics of various robotic competitions and how they compare to AI-DO Definitions of characteristics as they pertain to AI-DO are available in Table 2. A signifies that a competition currently possesses a characteristic.…”