LeDeepChef Deep Reinforcement Learning Agent for Families of Text-Based Games

Abstract

While Reinforcement Learning (RL) approaches lead to significant achievements in a variety of areas in recent history, natural language tasks remained mostly unaffected, due to the compositional and combinatorial nature that makes them notoriously hard to optimize. With the emerging field of Text-Based Games (TBGs), researchers try to bridge this gap. Inspired by the success of RL algorithms on Atari games, the idea is to develop new methods in a restricted game world and then gradually move to more complex environments. Previous work in the area of TBGs has mainly focused on solving individual games. We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme. In this work, we present our deep RL agent—LeDeepChef—that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions. The agent participated in Microsoft Research's First TextWorld Problems: A Language and Reinforcement Learning Challenge and outperformed all but one competitor on the final test set. The games from the challenge all share the same theme, namely cooking in a modern house environment, but differ significantly in the arrangement of the rooms, the presented objects, and the specific goal (recipe to cook). To build an agent that achieves high scores across a whole family of games, we use an actor-critic framework and prune the action-space by using ideas from hierarchical reinforcement learning and a specialized module trained on a recipe database.

Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games

“…To test our method's generalizability, we performed experiments on the cooking games considered in Adolphs and Hofmann (2019). A sample observation from these games looks like this: "You see a fridge.…”

“…AE-DQN (Action-Elimination DQN) -a combination of a Deep RL algorithm with an action eliminating network for sub-optimal actions -was proposed by Zahavy et al (Zahavy et al, 2018). Recent methods (Adolphs and Hofmann, 2019;Ammanabrolu and Riedl, 2018;Ammanabrolu and Hausknecht, 2020;Yin and May, 2019;Adhikari et al, 2020) use various heuristics to learn better state representations for efficiently solving complex TBGs.…”

How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

“…Text-based game-playing. introduce TextWorld, a framework for procedurally generating text-based games via grammars, and Yin and May, 2019;Adolphs and Hofmann, 2019;Adhikari et al, 2020) build agents that operate in this environment-focusing on aspects such as efficient exploration and zeroshot generalization to new, procedurally generated environments. Similarly, introduce Jericho, a framework and series of baseline agents for interacting with human-made textgames such as Zork (Anderson et al, 1979).…”

Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games

“…However, in reality, we usually hope an agent can learn a wide spectrum of tasks and generalize well to unseen environments. In (Adolphs and Hofmann, 2020), in terms of environments and task descriptions, researchers show that an actor-critic framework with action space pruning can learn an agent to generalize to unseen games that belongs to the same family when training. In this paper, with meta-reinforcement learning, we investigate if an agent can master multiple task types and generalize to unseen environments.…”