Humans feel too special for machines to score their morals

Abstract

Artificial Intelligence (AI) can be harnessed to create sophisticated social and moral scoring systems—enabling people and organizations to form judgements of others at scale. However, it also poses significant ethical challenges and is, subsequently, the subject of wide debate. As these technologies are developed and governing bodies face regulatory decisions, it is crucial that we understand the attraction or resistance that people have for AI moral scoring. Across four experiments, we show that the acceptability of moral scoring by AI is related to expectations about the quality of those scores, but that expectations about quality are compromised by people's tendency to see themselves as morally peculiar. We demonstrate that people overestimate the peculiarity of their moral profile, believe that AI will neglect this peculiarity, and resist for this reason the introduction of moral scoring by AI.

The Prospects and Challenges of Measuring Morality

“…Researchers should prioritize interpretable measures of morality. If people cannot understand how the moralometer generates its evaluations, it will be difficult to understand what the moralometer is capturing and to trust or challenge its results (Purcell & Bonnefon, 2023;von Eschenbach, 2021). One potential downside is that making the "answer key" available could help people learn how to look good on the test, but this can be mitigated by adhering to our fifth recommendation not to draw conclusions about individuals.…”