Advanced Skills through Multiple Adversarial Motion Priors in Reinforcement Learning

Abstract

In recent years, reinforcement learning (RL) has shown outstanding performance for locomotion control of highly articulated robotic systems. Such approaches typically involve tedious reward function tuning to achieve the desired motion style. Imitation learning approaches such as adversarial motion priors aim to reduce this problem by encouraging a pre-defined motion style. In this work, we present an approach to augment the concept of adversarial motion prior-based RL to allow for multiple, discretely switchable styles. We show that multiple styles and skills can be learned simultaneously without notable performance differences, even in combination with motion data-free skills. Our approach is validated in several real-world experiments with a wheeled-legged quadruped robot showing skills learned from existing RL controllers and trajectory optimization, such as ducking and walking, and novel skills such as switching between a quadrupedal and humanoid configuration. For the latter skill, the robot is required to stand up, navigate on two wheels, and sit down. Instead of tuning the sit-down motion, we verify that a reverse playback of the stand-up movement helps the robot discover feasible sit-down behaviors and avoids tedious reward function tuning.

Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations

“…This allows the learning agent to execute tasks that may not be portrayed in the original demonstrations. To enable active style control, Multi-AMP allows for the switching of multiple different style rewards by training multiple discriminators encoding different reference motions in parallel [39].…”

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

“…It can also build large-scale realistic complex scenes, and its underlying PhysX engine can accurately and realistically model and simulate the motion of objects. Therefore, more researchers have begun to use Isaac Gym as the implementation and verification platform of DRL algorithm [35][36][37][38] .…”

“…Lower-cost hardware platforms allow DRL algorithms to be more widely used. More recently, a wheel-legged quadruped robot [38] demonstrated skills learned from existing DRL controllers and trajectory optimization, such as ducking and walking, and new skills, such as switching between quadrupedal and humanoid configurations.…”

“…Many studies have also considered additional information, such as trajectory generators [46,[49][50][51] , control methods [52][53][54] , motion data [10,12,55,56] , etc. Trajectory generators and control methods mainly introduce prior knowledge in the action space, narrowing the search range of DRL control policies, which greatly improves the sample efficiency under a simple reward function.…”

“…Robots also struggle to generalize or adapt to other environments or tasks. To alleviate this problem to a certain extent, there have been recent DRL studies based on motion priors [86][87][88][89][90] , which have been successfully applied to quadrupedal locomotion tasks [12,56,91] . However, the variety of motion priors in these studies is insufficient, and the robot' s behavior is not agile and natural.…”