Computational Goals, Values and Decision-Making

Abstract

Considering the popular framing of an artificial intelligence as a rational agent that always seeks to maximise its expected utility, referred to as its goal, one of the features attributed to such rational agents is that they will never select an action which will change their goal. Therefore, if such an agent is to be friendly towards humanity, one argument goes, we must understand how to specify this friendliness in terms of a utility function. Wolfhart Totschnig (Fully Autonomous AI, Science and Engineering Ethics, 2020), argues in contrast that a fully autonomous agent will have the ability to change its utility function and will do so guided by its values. This commentary examines computational accounts of goals, values and decision-making. It rejects the idea that a rational agent will never select an action that changes its goal but also argues that an artificial intelligence is unlikely to be purely rational in terms of always acting to maximise a utility function. It nevertheless also challenges the idea that an agent which does not change its goal cannot be considered fully autonomous. It does agree that values are an important component of decision-making and explores a number of reasons why.

Utility Functions for Human/Robot Interaction

“…This paper gathers some preliminary ideas, and aims to serve as a "call-to-arms" to the community to examine edge cases in utility-based reasoning and ensure that they do not lead to paradoxical or undesirable behaviour. Significant avenues of future work remain open, including the integration of uncertainty into such utility-based systems, and we believe that philosophical work dealing with decision theory [16], as well as work on computational ethics [17], can serve to provide additional ideas to deal with the problem highlighted in this paper.…”

The Future Economics of Artificial Intelligence: Mythical Agents, a Singleton and the Dark Forest

“…Equivalents to utility functions 11 used in AI systems include value functions, objective functions, loss functions, reward functions (especially in Reinforcement Learning), and preference orderings (Eckersley, 2019). The concepts of utility function and goal are often used interchangeably in the AI literature (Dennis, 2020). Where loss functions (gradients) are used, which is the case when some objective function is minimized (for e.g.…”

“…Bounded rationality is not only applicable to humans, but also to AI agents -even though they may have vastly better computational abilities (Dennis, 2020). As Wagner (2020, p.114) point out "whilst the new species of 'machina economicus' [...] behaves more economic than man, it too is faced with bounded rationality.…”

“…The second cause of utility function instability is that an AI may change its own utility function autonomously (Dennis, 2020;Totschnig, 2020). It may do so to compensate for being boundedly rational.…”

“…Here AI developers have been exploring the programming of values, based on the argument that if the values of AI are aligned, then it will not change its goals in a direction that will be potentially harmful to humans. There is, however, at present no clear understanding of how to program this into AI systems: "our current lack of understanding about how to adequately program behaviour that can flexibly adopt and drop goals is one of the key limitations to our ability to take artificial intelligence to the next level" (Dennis, 2020(Dennis, , p.2493.…”