Realizing Learned Quadruped Locomotion Behaviors through Kinematic Motion Primitives

Abstract

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the "Stoch". D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.

Design, Development and Experimental Realization of A Quadrupedal Research Platform: Stoch

“…With a view toward realizing state of the art walking controllers in a low-cost walking platform, we developed a quadrupedal robot called the Stoch. This manuscript primarily serves as a detailed description of the hardware and software design process and construction of this platform (see Figure 1), which complements the results that were shown in [1]. This hardware features modular central body and a five bar co-axial leg design.…”

Learning Active Spine Behaviors for Dynamic and Efficient Locomotion in Quadruped Robots

“…It is the second generation robot in the Stoch series. A detailed description of the previous model can be found in [17], [20]. In this section, we first describe the overview of Stoch 2 robot and the essential hardware details, and then describe the simulation framework used for training.…”

“…B. State and Action Space 1) State Space: Similar to the work in [17], the state is represented by angles, velocities, torques of the active joints (legs and spine), and body orientation (in quaternions). The combined representation yields a 34-D state space.…”

“…We are employing Proximal Policy Optimization (PPO) [25] for learning optimal polices that yield action values in continuous space [17], [14]. The actor and critic network in the learning algorithm consists of two fully connected layers with the first and second layers consisting of 128 and 64 nodes respectively.…”

“…We have previously explored Deep Reinforcement learning (D-RL) based methods to realize efficient quadrupedal walking in Stoch [17], and our goal in this paper is to investigate the efficacy of D-RL for active spine behaviors. In other words, we want to revisit the problem of spine bounding with this new approach, which has seen a lot of success in recent years [14], [17], [18], [19]. Therefore, in this work, we provide insights into how RL can be successfully applied to such problems.…”