Few-shot Quality-Diversity Optimization

Abstract

In the past few years, a considerable amount of research has been dedicated to the exploitation of previous learning experiences and the design of Few-shot and Meta Learning approaches, in problem domains ranging from Computer Vision to Reinforcement Learning based control. A notable exception, where to the best of our knowledge, little to no effort has been made in this direction is Quality-Diversity (QD) optimisation. QD methods have been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. However, they remain costly due to their reliance on inherently sample inefficient evolutionary processes. We show that, given examples from a task distribution, information about the paths taken by optimisation in parameter space can be leveraged to build a prior population, which when used to initialise QD methods in unseen environments, allows for few-shot adaptation. Our proposed method does not require backpropagation. It is simple to implement and scale, and furthermore, it is agnostic to the underlying models that are being trained. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimisation in these environments.

Meta Neural Ordinary Differential Equations For Adaptive Asynchronous Control

“…Current work in RL/QD in general only handles one of those aspects, and robust methods for jointly addressing those problems are still lacking. For example, many systems that incorporate some notion of meta-learning [8,2,9], or system identification [3] are able to adapt to unseen tasks with minimal adaptation. However, to our knowledge, none of these methods address the problem of irregular actions and observations.…”