Reinforcement Learning with a Corrupted Reward Channel

Abstract

No real-world reward function is perfect. Sensory errors and software bugs may result in agents getting higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.

AGI Safety Literature Review

“…An AGI may be tempted to influence the data training its reward function so it points towards simple-to-optimize reward functions rather than harder ones (Armstrong, 2015). Everitt, Krakovna, et al (2017) show that the type of data the agent receives matter for reward learning corruption. In particular, if the reward data can be cross-checked between multiple sources, then the reward corruption incentive diminishes drastically.…”

Human‐centered artificial intelligence and machine learning

“…Commonsense failure goals occur when an intelligent agent does not achieve the desired result because part of the goal, or the way the goal should have been achieved, is left unstated (this is also referred to as a corrupted goal or corrupted reward; Everitt, Krakovna, Orseau, Hutter, & Legg, 2017). Why would this happen?…”

Defining the Scope of AI Regulations

“…Policy makers may want to address certain safety risks which are directly linked to reinforcement learning. This includes, among others, reward hacking [45,46] and interruptibility [47]. The following definition could be used:…”