How novelty search escapes the deceptive trap of learning to learn

Abstract

A major goal for researchers in neuroevolution is to evolve artificial neural networks (ANNs) that can learn during their lifetime. Such networks can adapt to changes in their environment that evolution on its own cannot anticipate. However, a profound problem with evolving adaptive systems is that if the impact of learning on the fitness of the agent is only marginal, then evolution is likely to produce individuals that do not exhibit the desired adaptive behavior. Instead, because it is easier at first to improve fitness without evolving the ability to learn, they are likely to exploit domaindependent static (i.e. non-adaptive) heuristics. This paper proposes a way to escape the deceptive trap of static policies based on the novelty search algorithm, which opens up a new avenue in the evolution of adaptive systems because it can exploit the behavioral difference between learning and nonlearning individuals. The main idea in novelty search is to abandon objective-based fitness and instead simply search only for novel behavior, which avoids deception entirely and has shown prior promising results in other domains. This paper shows that novelty search significantly outperforms fitness-based search in a tunably deceptive T-Maze navigation domain because it fosters the emergence of adaptive behavior.

Using a map-based encoding to evolve plastic neural networks

“…Figure 5 also shows that a diversity objective speeds up the search for solutions. These results are similar to the ones obtained by Risi [29] and Soltoggio [7]. In our experiment, using diversity enabled us to half the number of generation necessary to converge in many variants of these experiments.…”

“…This can be done through the use of diversity where an individual is compared to its nearest neighbors in the population, or using an archive of the previous generations [9], [28], in which case we talk about novelty. These methods have successfully been applied to the problem of evolving adaptive neural networks by Stanley and Risi [29] and Soltoggio [7] using novelty of behavior.…”

Robot coverage control by evolved neuromodulation

“…A number of these studies focus on neuromodulatory subsystems with projections that have diffuse action on the synapses of more classical neurons [36], [31], [34], [29]. On the other hand, this study develops a model of a single neuron (or homogenous nuclei) with dynamic regulation of learning and memory by an internal gene regulatory network.…”

Novelty-Based Multiobjectivization

“…Two recent papers [14,21] introduced a new but radical approach to evolve robot controllers: they propose to maximize the novelty of behaviors instead of the efficiency (the fitness) of these behaviors. This method, called "novelty search", relies on a user-defined distance between behaviors and the NEAT [22] evolutionary framework to synthesize neural networks.…”