Designing Preferences, Beliefs, and Identities for Artificial Intelligence

Abstract

Research in artificial intelligence, as well as in economics and other related fields, generally proceeds from the premise that each agent has a well-defined identity, well-defined preferences over outcomes, and well-defined beliefs about the world. However, as we design AI systems, we in fact need to specify where the boundaries between one agent and another in the system lie, what objective functions these agents aim to maximize, and to some extent even what belief formation processes they use.
 The premise of this paper is that as AI is being broadly deployed in the world, we need well-founded theories of, and methodologies and algorithms for, how to design preferences, identities, and beliefs. This paper lays out an approach to address these problems from a rigorous foundation in decision theory, game theory, social choice theory, and the algorithmic and computational aspects of these fields.

Achilles Heels for AGI/ASI via Decision Theoretic Adversaries

“…More recently, much work in AI alignment has fallen under the embedded agency paradigm. The process of understanding optimal and predictable behavior for agents embedded inside of an environment is complicated by conceptual challenges involving an agent's identity and world model [22,24]. Progress has been made through the formulation of Functional Decision Theory [68,18] which offers a framework for understanding optimal behavior in terms of having an optimal policy as opposed to making optimal choices.…”

Decision Propagation Networks for Image Classification

“…Belief propagation has been well studied for a long time especially by traditional methods [9,13]. Actually, the concept of belief propagation has also been exploited by various deep networks.…”

Local Justice and the Algorithmic Allocation of Scarce Societal Resources

“…It is certainly not a novel observation that we need to think carefully about what it is that we are trying to optimize. Just to give a couple of recent examples, Conitzer (2019) discusses the importance of appropriately designing preferences and optimization goals for AI agents, while a core argument of O'Neil and Gunn (2020) is that many of the problems of "near-term AI" (defined as expert systems that replace human decision-makers) are driven by a mismatch between the performance metrics of the AI (constructed by the algorithm designers) and the true objectives of stakeholders. Nevertheless, it is useful to get a sense of where the academic community has gone in response to these concerns.…”