Promoting Quality and Diversity in Population-based Reinforcement Learning via Hierarchical Trajectory Space Exploration

Miao, Jiayu; Zhou, Tianze; Shao, Kun; Zhou, Ming; Zhang, Weinan; Hao, Jianye; Yu, Yong; Wang, Jun

doi:10.1109/icra46639.2022.9811888

Cited by 1 publication

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By generating a diverse collection of controllers, we can endow a robot with a wide variety of useful behaviors, such as moving an arm into different configurations [1], [2], walking in every direction [3], [4], pushing an object through many trajectories [5], or traveling to any position in a maze [6], [7], [8]. Having a collection of controllers has proven particularly useful in robotic locomotion, where the collection enables a walking robot to adapt to damage [9], [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing

Tjanaka¹,

Fontaine²,

Aniruddha³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-training a diverse set of robot controllers in simulation has enabled robots to adapt online to damage in robot locomotion tasks. However, finding diverse, highperforming controllers requires specialized hardware and extensive tuning of a large number of hyperparameters. On the other hand, the Covariance Matrix Adaptation MAP-Annealing algorithm, an evolution strategies (ES)-based quality diversity algorithm, does not have these limitations and has been shown to achieve state-of-the-art performance in standard benchmark domains. However, CMA-MAE cannot scale to modern neural network controllers due to its quadratic complexity. We leverage efficient approximation methods in ES to propose three new CMA-MAE variants that scale to very high dimensions. Our experiments show that the variants outperform ES-based baselines in benchmark robotic locomotion tasks, while being comparable with state-of-the-art deep reinforcement learningbased quality diversity algorithms. Source code and videos are available at https://scalingcmamae.github.io Compute archive improvement Adapt and with ES Sample solutions Evaluate policies and insert into archive

show abstract