Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning

Abstract

We present the first complete attempt at concurrently training conversational agents that communicate only via self-generated language. Using DSTC2 as seed data, we trained natural language understanding (NLU) and generation (NLG) networks for each agent and let the agents interact online. We model the interaction as a stochastic collaborative game where each agent (player) has a role ("assistant", "tourist", "eater", etc.) and their own objectives, and can only interact via natural language they generate. Each agent, therefore, needs to learn to operate optimally in an environment with multiple sources of uncertainty (its own NLU and NLG, the other agent's NLU, Policy, and NLG). In our evaluation, we show that the stochastic-game agents outperform deep learning based supervised baselines.

Recent advances and challenges in task-oriented dialog systems

“…Then in the RL training phase, the dialog policy is alternately trained through learning from real users and planning with the environment model. Some other works jointly train a system policy and a user policy simultaneously [43,44].…”

Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition

“…Several studies have demonstrated that applying MARL delivers promising results in NLP tasks these years. While some methods use identical rewards for all agents (Das et al, 2017;Feng et al, 2018), other studies use completely separate rewards (Georgila et al, 2014;Papangelis et al, 2019). MADPL integrates two types of rewards by role-aware reward decomposition to train a better dialog policy in task-oriented dialog.…”

“…Two dialog agents interact with each other and collaborate to achieve the goal so that they require no explicit domain expertise, which helps develop a dialog system without the need of a well-built user simulator. Different from existing methods (Georgila et al, 2014;Papangelis et al, 2019), our approach is based on actor-critic framework (Barto et al, 1983) in order to facilitate pretraining and bootstrap the RL training. Following the paradigm of centralized training with decentralized execution (CTDE) (Bernstein et al, 2002) in multi-agent RL (MARL), the actor selects its action conditioned only on its local stateaction history, while the critic is trained with the actions of all agents.…”

IAI MovieBot

“…5 For the Telegram integration, the python-telegram-bot API is used. 6 The user's options, generated by the NLG, are shown as keyboard buttons in the Telegram app. The text of each button corresponds to a possible response and is linked to a speci c dialogue act.…”

“…OpenDial [5] is designed to facilitate the development of agents for single-turn Q&A-style dialogues. Plato [6] and PyDial [10] attempt to model user preferences, but do not track the preference evolution over conversations. Further, most of the available domain-speci c (movie) recommender systems are closed-source commercial products, such as the Facebook messenger bot And chill.…”