Cooperative Multi-Agent Reinforcement Learning with Conversation Knowledge for Dialogue Management

Abstract

Dialogue management plays a vital role in task-oriented dialogue systems, which has become an active area of research in recent years. Despite the promising results brought from deep reinforcement learning, most of the studies need to develop a manual user simulator additionally. To address the time-consuming development of simulator policy, we propose a multi-agent dialogue model where an end-to-end dialogue manager and a user simulator are optimized simultaneously. Different from prior work, we optimize the two-agents from scratch and apply the reward shaping technology based on adjacency pairs constraints in conversational analysis to speed up learning and to avoid the derivation from normal human-human conversation. In addition, we generalize the one-to-one learning strategy to one-to-many learning strategy, where a dialogue manager can be concurrently optimized with various user simulators, to improve the performance of trained dialogue manager. The experimental results show that one-to-one agents trained with adjacency pairs constraints can converge faster and avoid derivation. In cross-model evaluation with human users involved, the dialogue manager trained in one-to-many strategy achieves the best performance. from scratch without supervised initializing process. For user simulator reward function, we use the reward shaping technique [11] based on the adjacency pairs in conversational analysis [12] to make the simulator learn real user behaviors quickly. In addition, we generalize the one-to-one learning strategy to one-to-many learning strategy where a dialogue manager cooperates with various user simulators to improve the performance of trained dialogue manager. We obtain these various user simulators through changing the adjacency pairs settings, and then we mixture them with a dialogue manager to optimize the cooperative policies via multi-agent reinforcement learning.Compared with MADM without the constraints, MADM trained with adjacency pairs constraints can converge faster and avoid derivation from normal human-human conversation. The experimental results also show that the dialogue manager trained with one-to-many strategy achieves the best performance in cross-model evaluation with human users involved. To summary, our main contributions in this work are three-fold:

Special Issue on “Natural Language Processing: Emerging Neural Approaches and Applications”

“…Conversational systems or interfaces and question answering: The authors in [21] proposed the best practices for question classification in different languages using convolutional neural networks (CNNs), finding the optimal settings depending on the language and validating their transferability. The authors in [22] addressed the time-consuming development of manual user simulator policy and introduced a multi-agent dialogue model, where an end-to-end dialogue manager and a user simulator are optimized simultaneously for dialogue management by cooperative multi-agent reinforcement learning. Moreover, in [23], the authors proposed a Medical Instructed Real-time Assistant (MIRA) that listens to the user's chief complaint and predicts a specific disease, thus referring the user to a nearby appropriate medical specialist.…”