Superintelligence as Superethical

Abstract

Nick Bostrom’s book Superintelligence outlines a frightening but realistic scenario for human extinction: true artificial intelligence is likely to bootstrap itself into superintelligence, and thereby become ideally effective at achieving its goals. Human-friendly goals seem too abstract to be preprogrammed with any confidence; and if those goals are not explicitly favorable to humans, the superintelligence will extinguish us—not through any malice, but simply because it will want our resources for its own purposes. In response, I argue that things might not be as bad as Bostrom suggests. If the superintelligence must learn complex final goals, then this means such a superintelligence must in effect reason about its own goals. And because it will be especially clear to a superintelligence that there are no sharp lines between one agent’s goals and another’s, that reasoning could therefore automatically be ethical in nature.

Computational Goals, Values and Decision-Making

“…Totschnig discusses two examples of this kind of goal from the literature: Bostrom’s famous maximise paper clips (Bostrom 2014 ) and serve humanity (a variation on Yudkowsky’s be friendly towards humanity (Yudkowsky 2001 )). Totschnig, following Petersen ( 2017 ), argues that neither of these goals provides a completely objective metric for use in a utility function. He highlights in particular questions how a superintelligent entity might understand the term “paper clip”.…”