LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games

Abstract

While Reinforcement Learning (RL) approaches lead to significant achievements in a variety of areas in recent history, natural language tasks remained mostly unaffected, due to the compositional and combinatorial nature that makes them notoriously hard to optimize. With the emerging field of Text-Based Games (TBGs), researchers try to bridge this gap. Inspired by the success of RL algorithms on Atari games, the idea is to develop new methods in a restricted game world and then gradually move to more complex environments. Previous work in the area of TBGs has mainly focused on solving individual games. We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme. In this work, we present our deep RL agent-LeDeepChef -that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions. The agent participated in Microsoft Research's First TextWorld Problems: A Language and Reinforcement Learning Challenge and outperformed all but one competitor on the final test set. The games from the challenge all share the same theme, namely cooking in a modern house environment, but differ significantly in the arrangement of the rooms, the presented objects, and the specific goal (recipe to cook). To build an agent that achieves high scores across a whole family of games, we use an actor-critic framework and prune the action-space by using ideas from hierarchical reinforcement learning and a specialized module trained on a recipe database.

Towards Socially Intelligent Agents with Mental State Transition and Human Utility

“…Besides the LIGHT, there is also other text-adventure game frameworks, such as (Narasimhan et al, 2015) and TextWorld , but no human dialogues are incorporated in them. Based on the TextWorld, there are recent works Yin and May, 2019;Adolphs and Hofmann, 2019;Adhikari et al, 2020) on building agents trained with reinforce-ment learning. (Premack and Woodruff, 1978) defined Theory of Mind (ToM) as the ability to impute mental states to oneself and to others.…”

VisualHints: A Visual-Lingual Environment for Multimodal Reinforcement Learning

“…The action eliminating network is trained to predict invalid actions, supervised by an external elimination signal provided by the environment. More recent methods (Adolphs and Hofmann 2019;Ammanabrolu and Riedl 2018;Ammanabrolu and Hausknecht 2020;Yin and May 2019;Adhikari et al 2020) use different heuristics to learn better state representations for efficiently solving complex TBGs.…”

“…We use the cooking games used by Adolphs and Hofmann (2019) as the base games upon which we add the visual hints for multimodal RL. In order to implement the automatic and general addition of visual hints and textual clues to the existing textual observation space, we create an extra layer between the agent and the TextWorld game.…”

“…Observations are processed through two channels: the textual part consisting of context and command encoding, and the visual part which is responsible for encoding image features. The textual component is inspired from the work of Adolphs and Hofmann (2019).…”

“…We have also built a baseline agent that can use a multimodal approach to solve the game. This agent combines a textual module inspired by the work of Adolphs and Hofmann (2019) and a visual module that we have designed ourselves to analyze the floor plan. The main contributions of this paper are:…”