Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion

Li, Tianyu; Calandra, Roberto; Pathak, Deepak; Tian, Yuandong; Meier, Franziska; Rai, Akshara

doi:10.48550/arxiv.2008.11867

Cited by 3 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work has some parallels with model-based RL, as we do not rely on a full physics simulator or a physical robot to perform Monte Carlo rollouts. Recent work using model-based RL for legged robot control still needs to collect simulation data or physical robot data to learn either a transition model that contains the full state information [26] or center of mass (COM) information [27]. We rely instead on the simple centroidal dynamics to perform Monte Carlo rollouts, which does not require data from a full model or a physical robot.…”

Section: B Deep Reinforcement Learning For Quadrupedal Robotsmentioning

confidence: 99%

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

Xie¹,

Da²,

Babich³

et al. 2021

Preprint

View full text Add to dashboard Cite

Model-free reinforcement learning (RL) for legged locomotion commonly relies on a physics simulator that can accurately predict the behaviors of every degree of freedom of the robot. In contrast, approximate reduced-order models are often sufficient for many model-based control strategies. In this work we explore how RL can be effectively used with a centroidal model to generate robust control policies for quadrupedal locomotion. Advantages over RL with a full-order model include a simple reward structure, reduced computational costs, and robust sim-to-real transfer. We further show the potential of the method by demonstrating stepping-stone locomotion, twolegged in-place balance, balance beam locomotion, and sim-toreal transfer without further adaptations. Additional Results: https://www.pair.toronto.edu/glide-quadruped/.

show abstract

Section: B Deep Reinforcement Learning For Quadrupedal Robotsmentioning

confidence: 99%

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

Xie¹,

Da²,

Babich³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…From a design perspective, HRL allows for separate acquisition of low-level policies (options; skills), which can dramatically accelerate learning on downstream tasks. A variety of works propose the discovery of such low-level primitives from random walks [26,52], mutual information objectives [17,13,43,6], datasets of agent or expert traces [37,2,25], motion capture data [36,31,42], or from dedicated pre-training tasks [15,27].…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Skills for Efficient Exploration

Gehring¹,

Synnaeve²,

Krause³

et al. 2021

Preprint

View full text Add to dashboard Cite

In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest.In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for endto-end hierarchical reinforcement learning and unsupervised skill discovery. Code and videos are available at https://facebookresearch.github.io/hsd3.

show abstract

“…We present z i as scalar variables for clarity, but they can be generalized to higher-dimensional encoders, whose parameters and inputs are optimized using gradient descent and z i becomes the learned, fixed output of the z encoder. Such learned embeddings have been explored in literature before, for example in [20], [22], [18], [10]. However, none of these works generalize to multiple robot morphologies or use a high-dimensional image input as state.…”

Section: B Navigation Policies With Robot-specific Embeddingmentioning

confidence: 99%

Learning Navigation Skills for Legged Robots with Learned Robot Embeddings

Truong¹,

Yarats²,

Li³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Navigation policies are commonly learned on idealized cylinder agents in simulation, without modelling complex dynamics, like contact dynamics, arising from the interaction between the robot and the environment. Such policies perform poorly when deployed on complex and dynamic robots, such as legged robots. In this work, we learn hierarchical navigation policies that account for the low-level dynamics of legged robots, such as maximum speed, slipping, and achieve good performance at navigating cluttered indoor environments. Once such a policy is learned on one legged robot, it does not directly generalize to a different robot due to dynamical differences, which increases the cost of learning such a policy on new robots. To overcome this challenge, we learn dynamics-aware navigation policies across multiple robots with robot-specific embeddings, which enable generalization to new unseen robots. We train our policies across three legged robots -2 quadrupeds (A1, AlienGo) and a hexapod (Daisy). At test time, we study the performance of our learned policy on two new legged robots (Laikago, 4-legged Daisy) and show that our learned policy can sample-efficiently generalize to previously unseen robots.

show abstract

Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion

Cited by 3 publications

References 27 publications

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

Hierarchical Skills for Efficient Exploration

Learning Navigation Skills for Legged Robots with Learned Robot Embeddings

Contact Info

Product

Resources

About