Alignment for Advanced Machine Learning Systems

Abstract

This chapter surveys eight research areas organized around one question: As learning systems become increasingly intelligent and autonomous, what design principles can best ensure that their behavior is aligned with the interests of the operators? The chapter focuses on two major technical obstacles to AI alignment: the challenge of specifying the right kind of objective functions and the challenge of designing AI systems that avoid unintended consequences and undesirable behavior even in cases where the objective function does not line up perfectly with the intentions of the designers. The questions surveyed include the following: How can we train reinforcement learners to take actions that are more amenable to meaningful assessment by intelligent overseers? What kinds of objective functions incentivize a system to “not have an overly large impact” or “not have many side effects”? The chapter discusses these questions, related work, and potential directions for future research, with the goal of highlighting relevant research topics in machine learning that appear tractable today.

A Value-Sensitive Design Approach to Intelligent Agents

“…Disregarding emotion, it is fair to say that creators of intelligent agents would likely want emotion, again depending on the circumstance, to be left out, although design frameworks like the independent core observer model (ICOM) may ultimately prove more beneficial in the long run (Waser 2016). This is where adding to the counteridentical framework described above becomes critical: there are too many variables to account for and too many ways of solving or behaving during a situation that limiting certain decisions seem to be the only way, particularly to mitigate any adverse outcomes that may result from a top-down hierarchical goal structure (Taylor et al 2016b). That is unless the intelligent agent has transcended traditional artificial intelligence (Goertzel 2016;Rolf and Crook 2016).…”

Learning Norms from Stories

“…Humans have expectations that-just like other humans-agents will conform to personal values and to social norms [5], even when not explicitly communicated. This is the value alignment problem [1,4,26,30,33]. Some assert that agents should be imbued with the capability for moral decision making [10,32], but morals are more difficult to define than values or norms.…”

Transformational Communication via Evolving Ethical and Moral Norms of Lithuanian Civil Service Organizations

“…Furthermore, many civil service organizations try to encourage creativity and improve transparency; the lack of transparency, particularly in decision-making, is a key challenge among public administration organizations. Continuous learning and new knowledge application in technology might help obey human values and norms (Abel et al 2016) and reshape values and principles (Taylor et al 2017). Although it is difficult to cover a rich variety of complex knowledge domains by technology, due to tacitness of knowledge, Otterlo (2017) suggests formalizing present fundamentals of ethical codes via suitable computational logics, which could make ethical and moral norms-related communication more value-adding, engaging, and efficient.…”