Using Natural Language for Reward Shaping in Reinforcement Learning

Abstract

Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. However, designing appropriate shaping rewards is known to be difficult as well as time-consuming. In this work, we address this problem by using natural language instructions to perform reward shaping. We propose the LanguagE-Action Reward Network (LEARN), a framework that maps free-form natural language instructions to intermediate rewards based on actions taken by the agent. These intermediate language-based rewards can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari Learning Environment, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that, for the same number of interactions with the environment, language-based rewards lead to successful completion of the task 60% more often on average, compared to learning without language.

A Survey of Reinforcement Learning Informed by Natural Language

“…As illustrated in Figure 1, we separate the literature into language-conditional RL (in which interaction with language is necessitated by the problem formulation itself) and language-assisted RL (in which language is used to facilitate learning). The two categories are not mutually exclusive, in that for some languageconditional RL tasks, NLP methods or additional textual corpora are used to assist learning Goyal et al, 2019].…”

“…Methods developed for languageconditional tasks are relevant for language-assisted RL as they both deal with the problem of grounding natural language sentences in the context of RL. Moreover, in tasks such as following sequences of instructions, the full instructions are often not necessary to solve the underlying RL problem but they assist learning by structuring the policy [Andreas et al, 2017] or by providing auxiliary rewards [Goyal et al, 2019].…”

“…More recently, with the developments in deep learning, a common approach has been to embed both the instruction and observation to condition the policy directly [Mei et al, 2016;Hermann et al, 2017;Chaplot et al, 2018;Janner et al, 2018;Misra et al, 2017;. Human-generated natural language instructions are used in [MacMahon et al, 2006;Bisk et al, 2016;Misra et al, 2017;Janner et al, 2018;Anderson et al, 2018;Goyal et al, 2019;. Due to data-efficiency limitations of RL, this is not a standard in RL-based research [Hermann et al, 2017].…”

“…When environment rewards are available but sparse, instructions may still be used to generate auxiliary rewards to help learn efficiently. In this setting, [Goyal et al, 2019] and use auxiliary reward-learning modules trained offline to predict whether trajectory segments correspond to natural language annotations of expert trajectories. [Agarwal et al, 2019] perform a meta-optimisation to learn auxiliary rewards conditioned on features extracted from instructions.…”

“…Textual information can assist learning by specifying informative features, annotating states or entities in the environment, or describing subtasks in a multitask setting. In most cases covered here, the textual information is task-specific, with a few cases of using task-independent information through language parsers [Branavan et al, 2012] and pre-trained sentence embeddings [Goyal et al, 2019].…”