I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents

Abstract

Dialogue research tends to distinguish between chit-chat and goal-oriented tasks. While the former is arguably more naturalistic and has a wider use of language, the latter has clearer metrics and a straightforward learning signal. Humans effortlessly combine the two, for example engaging in chit-chat with the goal of exchanging information or eliciting a specific response. Here, we bridge the divide between these two domains in the setting of a rich multi-player text-based fantasy environment where agents and humans engage in both actions and dialogue. Specifically, we train a goal-oriented model with reinforcement learning against an imitation-learned "chit-chat" model with two approaches: the policy either learns to pick a topic or learns to pick an utterance given the top-K utterances from the chit-chat model. We show that both models outperform an inverse model baseline and can converse naturally with their dialogue partner in order to achieve goals.

How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

“…Visual and embodied agents using natural language instructions (Bisk et al, 2016;Kolve et al, 2017;Anderson et al, 2018) or in language-based action spaces (Das et al, 2017) utilize interactivity and environment grounding but have no notion of agent motivations, nor make any attempt to explicitly model commonsense reasoning. Perhaps closest in spirit to this work is Prabhumoye et al (2020), where they use artificially selected goals in LIGHT and train RL agents to achieve them. Similarly to the others, this work does not contain the motivations provided by LIGHT-Quests nor any modeling of commonsense reasoning.…”

“…The environment as seen in Figure 4 consists of three components. The first is a partner agent, which is a model trained to play other agents in the game, as in Prabhumoye et al (2020). Next is the game engine, which determines the effects of actions on the underlying game graph (Urbanek et al, 2019).…”

“…Following the format seen in Figure 3, the partner agent does not have a motivation itself but is trained to react to agents with motivations. Following Prabhumoye et al (2020), we keep the partner model fixed during the episodes where the LIGHT agent trains to ensure that it retains natural English semantics-avoiding the problem of language drift by learning an emergent language with that must agree with the partner's usage (Lee et al, 2019).…”