Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning

Abstract

Humans and animals are capable of quickly learning new behaviours to solve new tasks. Yet, we often forget that they also rely on a highly specialized morphology that co-adapted with motor control throughout thousands of years. Although compelling, the idea of co-adapting morphology and behaviours in robots is often unfeasible because of the long manufacturing times, and the need to redesign an appropriate controller for each morphology. In this paper, we propose a novel approach to automatically and efficiently co-adapt a robot morphology and its controller. Our approach is based on recent advances in deep reinforcement learning, and specifically the soft actor critic algorithm. Key to our approach is the possibility of leveraging previously tested morphologies and behaviors to estimate the performance of new candidate morphologies. As such, we can make full use of the information available for making more informed decisions, with the ultimate goal of achieving a more data-efficient co-adaptation (i.e., reducing the number of morphologies and behaviors tested). Simulated experiments show that our approach requires drastically less design prototypes to find good morphology-behaviour combinations, making this method particularly suitable for future co-adaptation of robot designs in the real world.

N-LIMB: Neural Limb Optimization for Efficient Morphological Design

“…Similar to our approach, several reinforcement learning-based strategies to co-optimization exist for both continuous [1,32,33,34] and discrete design spaces [10,35,14]. Luck et al [33] use a soft actor-critic algorithm and use a design-conditioned Q-function to evaluate designs.…”

“…Similar to our approach, several reinforcement learning-based strategies to co-optimization exist for both continuous [1,32,33,34] and discrete design spaces [10,35,14]. Luck et al [33] use a soft actor-critic algorithm and use a design-conditioned Q-function to evaluate designs. Chen et al [34] model the design space as a differentiable computational graph, which allows them to use standard gradient-based methods.…”