What Can Learned Intrinsic Rewards Capture?

Abstract

Reinforcement learning agents can include different components, such as policies, value functions, state representations, and environment models. Any or all of these can be the loci of knowledge, i.e., structures where knowledge, whether given or learned, can be deposited and reused. The objective of an agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. As far as the learning algorithm is concerned, these rewards are typically given and immutable. In this paper we instead consider the proposition that the reward function itself may be a good locus of knowledge. This is consistent with a common use, in the literature, of hand-designed intrinsic rewards to improve the learning dynamics of an agent. We adopt the multi-lifetime setting of the Optimal Rewards Framework, and propose to meta-learn an intrinsic reward function from experience that allows agents to maximise their extrinsic rewards accumulated until the end of their lifetimes. Rewards as a locus of knowledge provide guidance on "what" the agent should strive to do rather than "how" the agent should behave; the latter is more directly captured in policies or value functions for example. Thus, our focus here is on demonstrating the following: (1) that it is feasible to meta-learn good reward functions, (2) that the learned reward functions can capture interesting kinds of "what" knowledge, and (3) that because of the indirectness of this form of knowledge the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment. * Equal contribution. † Work done during an internship at DeepMind.

Adversarial Intrinsic Motivation for Reinforcement Learning

“…The optimal reward framework [60,63] and shaped rewards [47] (if generated by the agent itself) also consider intrinsic motivation as a way to assist an RL agent in learning the optimal policy for a given task. Such an intrinsically motivated reward signal has previously been learned through various methods such as evolutionary techniques [49,57], meta-gradient approaches [62,72,73], and others. The Wasserstein distance has been used to present a valid reward for imitation learning [70,17] as well as program synthesis [24].…”

“…Unfortunately, the optimal policy under such modified rewards might sometimes be different than the optimal policy under the task reward [47,16]. The problem of learning a reward signal that speeds up learning by communicating what to do but does not interfere by specifying how to do it is thus a useful and complex one [73].…”

Wasserstein Distance Maximizing Intrinsic Control

“…The optimal reward framework [33,35] and shaped rewards [23] (if generated by the agent itself) also consider intrinsic motivation as a way to assist an RL agent in learning the optimal policy for a given task. Such an intrinsically motivated reward signal has previously been learned through various methods such as evolutionary techniques [24,30], meta-gradient approaches [34,39,40], and others. The Wasserstein distance, in particular, has been used to present a valid reward for speeding up learning of goal-conditioned policies [11], imitation learning [37,10,38], as well as program synthesis [15].…”

Towards Continual Reinforcement Learning: A Review and Perspectives

“…Another core problem of continual RL that meta-learning can potentially help with is exploration. Meta-learning has been repeatedly used in the recent literature to learn functions for intrinsic motivation and improved exploration (Baranes and Oudeyer, 2009;Zheng et al, 2018;Xu et al, 2018a;Yang et al, 2019;Zou et al, 2019;Zheng et al, 2019).…”