Accelerated Policy Learning with Parallel Differentiable Simulation

Abstract

Deep reinforcement learning can generate complex control policies, but requires large amounts of training data to work effectively. Recent work has attempted to address this issue by leveraging differentiable simulators. However, inherent problems such as local minima and exploding/vanishing numerical gradients prevent these methods from being generally applied to control tasks with complex contact-rich dynamics, such as humanoid locomotion in classical RL benchmarks. In this work we present a high-performance differentiable simulator and a new policy learning algorithm (SHAC) that can effectively leverage simulation gradients, even in the presence of non-smoothness. Our learning algorithm alleviates problems with local minima through a smooth critic function, avoids vanishing/exploding gradients through a truncated learning window, and allows many physical environments to be run in parallel. We evaluate our method on classical RL control tasks, and show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms. In addition, we demonstrate the scalability of our method by applying it to the challenging high-dimensional problem of muscle-actuated locomotion with a large action space, achieving a greater than 17× reduction in training time over the best-performing established RL algorithm. More visual results are provided at: https://short-horizon-actor-critic.github.io/.

FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation

“…The proposed tasks enabled us to study manipulation problems with fluids comprehensively, and shed light on challenges of existing methods, and the potential of utilizing differentiable physics as an effective tool for trajectory optimization in the context of complex fluid manipulation. One future direction is to extend towards more realistic problem setups using visual input, and to distill policy optimized using differentiable physics into neural-network based policies (Lin et al, 2022a;Xu et al, 2023), or use gradients provided by differentiable simulation to guide policy learning (Xu et al, 2022). Furthermore, since MPM was originally proposed, researchers have been using it to simulate various realistic materials with intricate properties, such as snow, sand, mud, and granular materials.…”

“…Another approach is to implement physics simulations in a differentiable manner, usually with automatic differentiation tools (Hu et al, 2019a;de Avila Belbute-Peres et al, 2018;Xu et al, 2022;Huang et al, 2021), and the gradient information provided by these simulations has shown to be helpful in control and manipulation tasks concerning rigid and soft bodies (Xu et al, 2021;Lin et al, 2022a;Xu et al, 2022;Li et al, 2022;Wang et al, 2023). On a related note, Taichi (Hu et al, 2019b) and Nvidia Warp (Macklin, 2022) are two recently proposed domain-specific programming language for GPU-accelerated simulation.…”

“…We evaluate state-of-the-art Reinforcement Learning algorithms and trajectory optimization methods in FluidLab, and highlight major challenges with existing methods for solving the proposed tasks. In addition, since differentiable physics has proved useful in many rigid-body and soft-body robotic tasks (Xu et al, 2021;Lin et al, 2022a;Xu et al, 2022;Li et al, 2022), it is desirable to extend it to fluid manipulation tasks. However, supporting such a wide spectrum of materials present in FluidLab in a differentiable way is extremely challenging, and optimization using the gradients is also difficult due to the highly non-smooth optimization landscapes of the proposed tasks.…”

New subspace minimization conjugate gradient methods based on regularization model for unconstrained optimization

“…This paper delves into the model-based setting (Chua et al, 2018;Janner et al, 2019;Kaiser et al, 2019;Zhang, 2022;Hafner et al, 2023), where a learned model is employed to train a control policy. Recent approaches (Mora et al, 2021;Suh et al, 2022a,b;Xu et al, 2022) based on differentiable simulators (Freeman et al, 2021;Heiden et al, 2021b) assume that gradients of simulation outcomes w.r.t. actions are explicitly given.…”

“…When the model is good enough, a small h may not fully leverage the accurate gradient information. As evidence, approaches (Xu et al, 2022;Mora et al, 2021) based on differentiable simulators typically adopt longer unrolls compared to model-based approaches. Therefore, with SN, more accurate multi-step predictions should enable more efficient learning without making the underlying optimization process harder.…”