Learning Behavior Characterizations for Novelty Search

Abstract

Novelty search and related diversity-driven algorithms provide a promising approach to overcoming deception in complex domains. The behavior characterization (BC) is a critical choice in the application of such algorithms. The BC maps each evaluated individual to a behavior, i.e., some vector representation of what the individual is or does during evaluation. Search is then driven towards diversity in a metric space of these behaviors. BCs are built from handdesigned features that are limited by human expertise, or upon generic descriptors that cannot exploit domain nuance. The main contribution of this paper is an approach that addresses these shortcomings. Generic behaviors are recorded from evolution on several training tasks, and a new BC is learned from them that funnels evolution towards successful behaviors on any further tasks drawn from the domain. This approach is tested in increasingly complex simulated mazesolving domains, where it outperforms both hand-coded and generic BCs, in addition to outperforming objective-based search. The conclusion is that adaptive BCs can improve search in many-task domains with little human expertise.

Autonomous skill discovery with quality-diversity and unsupervised descriptors

“…Meyerson et al [23] present a method that enables the automatic determination of BD by learning which descriptor usually leads to controllers that e ectively solve the task. ey rst use some existing BDs with a default QD algorithm (novelty search) to train a population of controllers on di erent "training" tasks.…”

Being curious about the answers to questions: novelty search with learned attention

“…If we wished to make an agent that could answer any one of a set of questions which might be posed to it (similar to ideas of multi-task behavior characterizations in Meyerson et al (2016)), we could ask an agent to fill its memory with Figure 1: General schematic of an inference network which also learns a behavior characterization and can evaluate the saliency of a given proposed action to objectives such as novelty search or question-motivated curiosity. The dashed lines represent optional information flows that can be added without disrupting the ability to correctly embed action proposals.…”

“…In novelty search, this takes the form of the behavior characterization -the way in which distinct agent behaviors or outcomes are embedded into a Euclidean space in order to assess their novelty. While much of the work uses hand-crafted behavior characterizations based on some knowledge of the relevant degrees of freedom of the task at hand, there has been work in formulating general characterizations which work across many tasks (Doncieux and Mouret, 2013), and in learning the characterizations with respect to specific criteria for the quality of exploration (Meyerson et al, 2016).…”

Enhanced optimization with composite objectives and novelty pulsation

“…Another version is to accept novel solutions only if they satisfy minimal performance criteria [17,32]. Some of these approaches have been generalized using the idea of behavior domination to discover stepping stones [34,35].…”