Computer Says I Don’t Know: An Empirical Approach to Capture Moral Uncertainty in Artificial Intelligence

Abstract

As AI Systems become increasingly autonomous, they are expected to engage in decision-making processes that have moral implications. In this research we integrate theoretical and empirical lines of thought to address the matters of moral reasoning and moral uncertainty in AI Systems. We reconceptualize the metanormative framework for decision-making under moral uncertainty and we operationalize it through a latent class choice model. The core idea being that moral heterogeneity in society can be codified in terms of a small number of classes with distinct moral preferences and that this codification can be used to express moral uncertainty of an AI. Choice analysis allows for the identification of classes and their moral preferences based on observed choice data. Our reformulation of the metanormative framework is theory-rooted and practical in the sense that it avoids runtime issues in real time applications. To illustrate our approach we conceptualize a society in which AI Systems are in charge of making policy choices. While one of the systems uses a baseline morally certain model, the other uses a morally uncertain model. We highlight cases in which the AI Systems disagree about the policy to be chosen, thus illustrating the need to capture moral uncertainty in AI systems.

Models of moral decision making: Theory and empirical applications in various domains

“…We hope that this special issue helps encourage the choice modelling community to increasingly devote attention to moral decision making and to maintain a cross-disciplinary view that takes on board progress in fields as diverse as (experimental) ethics and moral psychology, in addition to insights from the economics discipline. Moral decision making is a worthy topic indeed, and one which is likely to become even more relevant in years to come, as moral choice models are becoming an obvious candidate to equip artificial agents with a human-inspired moral compass (Noothigattu et al, 2018;Feier et al 2021;Martinho et al, 2021).…”

Moral Uncertainty and the Problem of Fanaticism

“…As emphasised in (Gabriel 2020), given the diversity of ethical perspectives, an agent that acts according to a single ethical theory may not receive societal endorsement, thus jeopardising the project of machine ethics. Indeed, recent work in machine ethics draws on insights from moral uncer-tainty (Bogosian 2017;Bhargava and Kim 2017;Martinho, Kroesen, and Chorus 2021;Dobbe, Gilbert, and Mintz 2020;Ecoffet and Lehman 2021), and advocate for the identification of principles for the fair aggregation of various ethical perspectives. Note that, despite the aforementioned issues, all existing work lack a formal definition of fanaticism.…”