Learning Dialog Policies from Weak Demonstrations

Abstract

Deep reinforcement learning is a promising approach to training a dialog manager, but current methods struggle with the large state and action spaces of multi-domain dialog systems. Building upon Deep Q-learning from Demonstrations (DQfD), an algorithm that scores highly in difficult Atari games, we leverage dialog data to guide the agent to successfully respond to a user's requests. We make progressively fewer assumptions about the data needed, using labeled, reduced-labeled, and even unlabeled data to train expert demonstrators. We introduce Reinforced Fine-tune Learning, an extension to DQfD, enabling us to overcome the domain gap between the datasets and the environment. Experiments in a challenging multi-domain dialog system framework validate our approaches, and get high success rates even when trained on outof-domain data.

A survey on deep reinforcement learning for audio-based applications

“…They train the proposed SDS using a network of DQN agents, which is similar to hierarchical DRL but with more flexibility for transitioning across dialogues domains. Another work-related to faster training is proposed by Gordon-Hall et al (2020), where the behaviour of RL agents is guided by expert demonstrations.…”

Enhancing Designer Knowledge to Dialogue Management: A Comparison between Supervised and Reinforcement Learning Approaches

“…[Nishimoto and Reali Costa 2019] extended the first work by showing that a good balance in exploration and exploitation during training can significantly improve the performance. Some other recent works also used the classical DQN algorithm to train the policy [Gordon-Hall et al 2020, Wang et al 2020, showing that despite simple, this algorithm can provide good results [Mo et al 2018] and [Weisz et al 2018] tried out other RL algorithms to model the DM, such as SARSA and actor-critic, respectively. Finally, [Saha et al 2020] proposed a hierarchical deep reinforcement learning approach to deal with more complex dialogue systems and [Takanobu et al 2019] proposed a method to learn the reward and optimize the policy jointly.…”

“…However, it is more complicated and needs a lot of labeled data collected from experts. [Gordon-Hall et al 2020] proposed the Deep Q-learning from Demonstrations (DQfD), which uses expert demonstrators in a weakly supervised fashion.…”

“…In order to compare the results to a baseline, we adopt the same parameters' values from [Nishimoto and Reali Costa 2019] work. Although its a simple algorithm, there are still many works that use DQN (or some variant) to train the DM [Li et al 2017, Nishimoto and Reali Costa 2019, Gordon-Hall et al 2020, Wang et al 2020. For this reason we used the DQN algorithm and focused in improving the warm-up phase.…”

“…End-to-End User Many works focus in the pipeline architecture, specially in the policy component of the DM module. In brief, they employ rule-based DM, supervised learning [Vlasov et al 2019, Hosseini-Asl et al 2020; and reinforcement learning [Saha et al 2020, Wang et al 2020, Gordon-Hall et al 2020.…”