Discovering Generalizable Skills via Automated Generation of Diverse Tasks

Abstract

The learning efficiency and generalization ability of an intelligent agent can be greatly improved by utilizing a useful set of skills. However, the design of robot skills can often be intractable in real-world applications due to the prohibitive amount of effort and expertise that it requires. In this work, we introduce Skill Learning In Diversified Environments (SLIDE), a method to discover generalizable skills via automated generation of a diverse set of tasks. As opposed to prior work on unsupervised discovery of skills which incentivizes the skills to produce different outcomes in the same environment, our method pairs each skill with a unique task produced by a trainable task generator. To encourage generalizable skills to emerge, our method trains each skill to specialize in the paired task and maximizes the diversity of the generated tasks. A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective. The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks. We demonstrate that the proposed method can effectively learn a variety of robot skills in two tabletop manipulation domains. Our results suggest that the learned skills can effectively improve the robot's performance in various unseen target tasks compared to existing reinforcement learning and skill learning methods.

Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey

“…Kovač et al (2020) proposed to enhance these methods with a goal sampling prior focusing goal selection towards controllable areas of the goal space. Finally, Fang et al (2021) use procedural content generation (pcg) to train a task generator that produces diverse environments in which agents can explore customized skills.…”

PORTAL: Automatic Curricula Generation for Multiagent Reinforcement Learning

“…Some works consider the learning relevance of a task to the final task, but they also have limitations. The approach in (Fang et al 2021) can only work with the assumption that the tasks have the same state and action space, and thus could not be applied in complex MASs. The approaches in (da Silva and Costa 2018; Zhao and Pajarinen 2022) need to work with task contexts, like object type and number, goal position, etc.…”