Efficient Novelty Search Through Deep Reinforcement Learning

Abstract

Novelty search, which was inspired by the nature that evolves creatures with diversity, has shown great potential in solving reinforcement learning (RL) tasks with sparse and deceptive rewards. However, most of the existing novelty search methods evolve the populations through hybrization and mutation, which is inefficient in diverging populations. In this paper, we propose a method which incorporates deep RL with novelty search to improve the efficiency of diverging the populations for novelty search. We first propose a strategy that improves the novelty of individuals generated by genetic algorithm using reinforcement learning. Based on this strategy, we propose a framework that incorporates deep RL with novelty search, and then derive an algorithm to improve the search efficiency of the novelty search for continuous control tasks. Our experimental results show that our method can improve the search efficiency of novelty search and can also provide a competitive performance compared to some of the existing novelty search methods. The

Few-shot Quality-Diversity Optimization

“…In a similar manner, ME-ES [24] leverages the more efficient ES strategies to replace the random exploration of vanilla MAP-elites. In other recent works [25], in addition to selecting and mutating most novel individuals and in order to accelerate behavior space coverage, the authors propose to push less novel individuals toward the most novel ones by creating targets in behavior space, in a manner reminiscent of Goal Exploration Processes [26].…”