Reinforcement Learning Meets Hybrid Zero Dynamics: A Case Study for RABBIT

Abstract

The design of feedback controllers for bipedal robots is challenging due to the hybrid nature of its dynamics and the complexity imposed by high-dimensional bipedal models. In this paper, we present a novel approach for the design of feedback controllers using Reinforcement Learning (RL) and Hybrid Zero Dynamics (HZD). Existing RL approaches for bipedal walking are inefficient as they do not consider the underlying physics, often requires substantial training, and the resulting controller may not be applicable to real robots. HZD is a powerful tool for bipedal control with local stability guarantees of the walking limit cycles. In this paper, we propose a non traditional RL structure that embeds the HZD framework into the policy learning. More specifically, we propose to use RL to find a control policy that maps from the robot's reduced order states to a set of parameters that define the desired trajectories for the robot's joints through the virtual constraints. Then, these trajectories are tracked using an adaptive PD controller. The method results in a stable and robust control policy that is able to track variable speed within a continuous interval. Robustness of the policy is evaluated by applying external forces to the torso of the robot. The proposed RL framework is implemented and demonstrated in OpenAI Gym with the MuJoCo physics engine based on the well-known RABBIT robot model.

Robust Feedback Motion Policy Design Using Reinforcement Learning on a 3D Digit Bipedal Robot

“…In this section, we build upon our previous work proposed in [11], [13] to implement a cascade-structure learning framework that realizes stable and robust walking gaits for the 3D bipedal robots. The specific design ensures successful transferring learned policies in simulation to robot hardware with minimal turning.…”

“…Foot placement regulation controller has been widely used in 3D bipedal walking robots with the objective of improving the speed tracking and the stability and robustness of the walking gait [20]- [22]. Longitudinal speed regulation, defined by (13), sets a target offset in the swing hip pitch joint, whereas lateral speed regulation (11) do the same for the swing hip roll angle. Direction regulation (12) add an offset to the yaw hip angle to keep the torso yaw orientation at the desired angle.…”

“…[8], [9] learned the Poincaré map of the periodic walking pattern and applied the method to two 2D bipedal robots. Some recent work has proposed to learn the joint-level trajectory for each joint as the reference motion through supervised learning [10] or using reinforcement learning [11]- [13]. This approach simplifies the design of the lower-level tracking, which can be as simple as a PD controller.…”

“…Motivated by the existing challenges, this paper enhances the hybrid zero dynamics inspired reinforcement learning framework presented in our previous work [11], [13] emphasizing the policy robustness and sim-to-real transfer. The proposed cascade control structure couples the high-level learning-based controller with the low-level model-based controller.…”

Preference-Based Learning for User-Guided HZD Gait Generation on Bipedal Walking Robots

“…To circumvent this engineering empiricism, the field of machine learning has approached bipedal locomotion from many perspectives, including reinforcement leaning and imitation learning. Reinforcement learning simplifies the process of "learning to walk" [13] without prior knowledge [14]- [17], but because this method relies on a carefully crafted reward function, the behavior is exclusively determined by its construction. This motivates the second method, imitation learning, which infers the underlying reward function from expert demonstrations [18]- [20].…”