Risks from Learned Optimization in Advanced Machine Learning Systems

Abstract

We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer-a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be-how will it differ from the loss function it was trained under-and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.

Alignment of Language Agents

“…Of particular concern is when an agent is optimizing for the wrong thing when out of distribution. Hubinger et al (2019) introduce the concept of a mesa-optimizer -a learnt model which is itself an optimizer for some mesa-objective, which may differ from the base-objective used to train the model, when deployed outside of the training environment. This leads to the so-called inner alignment problem:…”

“…Of particular concern is deceptive alignment (Hubinger et al, 2019), where the mesa-optimizer acts as if it's optimizing the base objective as an instrumental goal, whereas its actual mesa-objective is different.…”

“…One suggestion for addressing the inner alignment problem involves using interpretability tools for evaluating and performing adversarial training (Hubinger, 2019). There are a number of works on interpretability and analysis tools for NLP, see for example the survey of Belinkov and Glass (2019).…”

Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity

“…Understanding such levels gives insight into how open-ended search can diverge from the system designer's intents, creating potential safety hazards. Note that the categorization presented next is adapted from previous AI safety categorizations (Ortega and Maini, 2018;Hubinger et al, 2019).…”

“…From this point of view, nearly everything of moral worth results from humanity transcending the explicit incentives of the search algorithm. In contrast, a central focus within top-down AI safety is to explicitly align an AI's incentives with our own (Hubinger et al, 2019;Taylor et al, 2016a), e.g. by modeling human preferences to use as an objective function (Leike et al, 2018), or to be cautious of divergences between explicit incentives and agent incentives (Hubinger et al, 2019).…”