Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

Wang, Rui; Lehman, Joel; Rawal, Aditya; Zhi, Jiale; Li, Yulun; Clune, Jeff; Stanley, Kenneth O.

doi:10.48550/arxiv.2003.08536

Cited by 6 publications

(8 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work from Domain Randomization [31] is not applicable to learned simulators because they often do not have easily configurable parameters. Future direction for this work could be modifying the dynamics model parameters in a targeted manner [28,33,34]. This simple approach to generating different versions of a model could also be useful in committee-based methods [25,26].…”

Section: Discussionmentioning

confidence: 99%

Dropout's Dream Land: Generalization from Learned Simulators to Reality

Wellmer¹,

Kwok²

2021

Preprint

View full text Add to dashboard Cite

A World Model is a generative model used to simulate an environment. World Models have proven capable of learning spatial and temporal representations of Reinforcement Learning environments. In some cases, a World Model offers an agent the opportunity to learn entirely inside of its own dream environment. In this work we explore improving the generalization capabilities from dream environments to real environments (Dream2Real). We present a general approach to improve a controller's ability to transfer from a neural network dream environment to reality at little additional cost. These improvements are gained by drawing on inspiration from Domain Randomization, where the basic idea is to randomize as much of a simulator as possible without fundamentally changing the task at hand. Generally, Domain Randomization assumes access to a pre-built simulator with configurable parameters but oftentimes this is not available. By training the World Model using dropout, the dream environment is capable of creating a nearly infinite number of different dream environments. Previous use cases of dropout either do not use dropout at inference time or averages the predictions generated by multiple sampled masks (Monte-Carlo Dropout). Dropout's Dream Land leverages each unique mask to create a diverse set of dream environments. Our experimental results show that Dropout's Dream Land is an effective technique to bridge the reality gap between dream environments and reality. Furthermore, we additionally perform an extensive set of ablation studies. 1

show abstract

Section: Discussionmentioning

confidence: 99%

Dropout's Dream Land: Generalization from Learned Simulators to Reality

Wellmer¹,

Kwok²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To create a curriculum, the threshold is linearly increased over the course of training. POET [45,46] uses a population of minimax (rather than minimax regret) adversaries to generate the terrain for a 2D walker. However, POET [45] requires generating many new environments, testing all agents within each one, and discarding environments based on a manually chosen reward threshold, which wastes a significant amount of computation.…”

Section: Related Workmentioning

confidence: 99%

“…Prior work [45,46] focused on demonstrating emergent complexity as the primary goal, arguing that automatically learning complex behaviors is key to improving the sophistication of AI agents. Here, we track the complexity of the generated environments and learned behaviors throughout training.…”

Section: Emergent Complexitymentioning

confidence: 99%

“…Many reinforcement learning problems require designing a distribution of tasks and environments that can be used to evaluate and train effective policies. This is true for a diverse array of methods including transfer learning (e.g., [2,48,32,41]), robust RL (e.g., [4,16,28]), unsupervised RL (e.g., [15]), and emergent complexity (e.g., [38,45,46]). For example, suppose we wish to train a robot in simulation to pick up objects from a bin in a real-world warehouse.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

Dennis,

Jaques,

Vinitsky

et al. 2020

Preprint

View full text Add to dashboard Cite

A wide range of reinforcement learning (RL) problems -including robustness, transfer learning, unsupervised RL, and emergent complexity -require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments. * Equal contribution Preprint. Under review.

show abstract

“…In part because of their natural ability to deal with those issues, Quality-Diversity [1] (QD) methods have garnered considerable interest in the past few years, and have been applied to many problems, ranging from robotics [2], [3], [4], [5], [6], [7] to video games [8], [9], [10] and open-ended learning [11], [12]. The aim of these methods is to obtain a diverse set of well-performing solutions to a given problem.…”

Section: Introductionmentioning

confidence: 99%

Few-shot Quality-Diversity Optimization

Salehi,

Coninx,

Doncieux

2021

Preprint

View full text Add to dashboard Cite

In the past few years, a considerable amount of research has been dedicated to the exploitation of previous learning experiences and the design of Few-shot and Meta Learning approaches, in problem domains ranging from Computer Vision to Reinforcement Learning based control. A notable exception, where to the best of our knowledge, little to no effort has been made in this direction is Quality-Diversity (QD) optimisation. QD methods have been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. However, they remain costly due to their reliance on inherently sample inefficient evolutionary processes. We show that, given examples from a task distribution, information about the paths taken by optimisation in parameter space can be leveraged to build a prior population, which when used to initialise QD methods in unseen environments, allows for few-shot adaptation. Our proposed method does not require backpropagation. It is simple to implement and scale, and furthermore, it is agnostic to the underlying models that are being trained. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimisation in these environments.

show abstract

Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

Cited by 6 publications

References 43 publications

Dropout's Dream Land: Generalization from Learned Simulators to Reality

Dropout's Dream Land: Generalization from Learned Simulators to Reality

Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

Few-shot Quality-Diversity Optimization

Contact Info

Product

Resources

About