The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities

Lehman, Joel; Clune, Jeff; Misevic, Dusan; Adami, Christoph; Beaulieu, Julie; Bentley, P. J.; Bernard, Samuel; Beslon, Guillaume; Bryson, David M.; Cheney, Nick; Cully, Antoine; Doncieux, Stéphane; Dyer, Fred C.; Ellefsen, Kai Olav; Feldt, Robert; Fischer, Stephan; Forrest, Stephanie; Frénoy, Antoine; Gagneé, Christian; Goff, Léni Le; Grabowski, Laura M.; Hodjat, Babak; Keller, Laurent; Knibbe, Carole; Krčah, Peter; Lenski, Richard E.; Lipson, Hod; MacCurdy, Robert; Maestre, Carlos; Miikkulainen, Risto; Mitri, Sara; Moriarty, David E.; Mouret, Jean-Baptiste; Nguyen, Anh; Ofria, Charles; Parizeau, Marc; Parsons, David; Pennock, Robert T.; Punch, William F.; Ray, Thomas S.; Schoenauer, Marc; Shulte, Eric; Sims, Karl; Stanley, Kenneth O.; Taddéi, François; Tarapore, Danesh; Thibault, Simon; Weimer, Westley; Watson, Richard H.; yosinksi, Jason

doi:10.48550/arxiv.1803.03453

Cited by 30 publications

(41 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The success of reward modeling relies heavily on the quality of the reward model. If the reward model only captures most aspects of the objective but not all of it, this can lead the agent to find undesirable degenerate solutions Lehman et al, 2018;Ibarz et al, 2018). In other words, the agent's behavior depends on the reward model in a way that is potentially very fragile.…”

Section: Challengesmentioning

confidence: 99%

“…On the one hand, we want ML to generate creative and brilliant solutions like AlphaGo's Move 37 (Metz, 2016)-a move that no human would have recommended, yet it completely turned the game in AlphaGo's favor. On the other hand, we want to avoid degenerate solutions that lead to undesired behavior like exploiting a bug in the environment simulator (Clark & Amodei, 2016;Lehman et al, 2018). In order to differentiate between these two outcomes, our agent needs to understand its user's intentions, and robustly achieve these intentions with its behavior.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scalable agent alignment via reward modeling: a research direction

Leike,

Krueger,

Everitt

et al. 2018

Preprint

View full text Add to dashboard Cite

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents. * Work done during an internship at DeepMind.

show abstract

Section: Challengesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Scalable agent alignment via reward modeling: a research direction

Leike,

Krueger,

Everitt

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Motivated by difficulty in reward specification [21], inverse reinforcement learning (IRL) methods estimate a reward function from human demonstrations [24,1,20,17,23]. The central assumption behind these methods is that human behavior is rational, i.e., optimal with respect to their reward (cumulative, in expectation).…”

Section: Introductionmentioning

confidence: 99%

Human irrationality: both bad and good for reward inference

Chan¹,

Critch²,

Dragan³

2021

Preprint

View full text Add to dashboard Cite

Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior. But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference. The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization. We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference.We find that wrongly modeling a systematically irrational human as noisy-rational performs a lot worse than correctly capturing these biases -so much so that it can be better to skip inference altogether and stick to the prior! More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can. That is, if a robot has the correct model of a human's irrationality, it can make an even stronger inference than it ever could if the human were rational. Irrationality fundamentally helps rather than hinder reward inference, but it needs to be correctly accounted for. We explain this theoretically by showing that irrationality can increase the mutual information between the policy and the reward. We also replicate the result in an autonomous driving domain with myopic/greedy demonstrations.

show abstract

“…EC methods were chosen because EC is arguably the most versatile of the metalearning approaches. It is a population-based search method; allowing for extensive exploration, which often results in creative, novel solutions that are not obvious at first [16]. EC has been successful in hyperparameter optimization and architecture design in particular [18,26,22,17].…”

Section: Introductionmentioning

confidence: 99%

Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization

González¹,

Miikkulainen²

2019

Preprint

Self Cite

View full text Add to dashboard Cite

As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss-function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.Preprint. Under review.

show abstract

The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities

Cited by 30 publications

References 0 publications

Scalable agent alignment via reward modeling: a research direction

Scalable agent alignment via reward modeling: a research direction

Human irrationality: both bad and good for reward inference

Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization

Contact Info

Product

Resources

About