DeepQ Stepper: A framework for reactive dynamic walking on uneven terrain

Abstract

Reactive stepping and push recovery for biped robots is often restricted to flat terrains because of the difficulty in computing capture regions for nonlinear dynamic models. In this paper, we address this limitation by using reinforcement learning to approximately learn the 3D capture region for such systems. We propose a novel 3D reactive stepper, The DeepQ stepper, that computes optimal step locations for walking at different velocities using the 3D capture regions approximated by the action-value function. We demonstrate the ability of the approach to learn stepping with a simplified 3D pendulum model and a full robot dynamics. Further, the stepper achieves a higher performance when it learns approximate capture regions while taking into account the entire dynamics of the robot that are often ignored in existing reactive steppers based on simplified models. The DeepQ stepper can handle non convex terrain with obstacles, walk on restricted surfaces like stepping stones and recover from external disturbances for a constant computational cost.

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

“…This limits our ability to deal with more challenging terrain, e.g., more challenging variations of the stepping stone task. Incorporating systems [36,44] that directly learn foot placement will be crucial for further improving the capability and robustness of our system.…”

“…Heuristics or dynamics can be learned to enable faster optimization or planning, e.g., [18,33]. Hierarchical control structures have been proposed to allow model-based control and model-free RL policies to operate at different time scales to leverage their respective advantages, e.g., [4,34,35,36].…”

Safe Value Functions

“…The penalty is then seen as a design choice controlling the risk of optimal controllers [21], or as a parameter that must be optimized over to allow for the least conservative safe behaviour [22], [23]. In practice, the penalty is often treated as a heuristic [24]- [27]. From a theoretical point of view, it is unclear whether optimal controllers of penalized problems enjoy safety guarantees, and whether they are optimal with respect to the original constrained task [2], [28].…”