Audio Embeddings Help to Learn Better Dialogue Policies

Abstract

Neural transformer architectures have gained a lot of interest for text-based dialogue management in the last few years. They have shown high learning capabilities for open domain dialogue with huge amounts of data and also for domain adaptation in task-oriented setups. But the potential benefits of exploiting the users' audio signal have rarely been explored in such frameworks. In this work, we combine text dialogue history representations generated by a GPT-2 model with audio embeddings obtained by the recently released Wav2Vec2 transformer model. We jointly fine-tune these models to learn dialogue policies via supervised learning and two policy gradient-based reinforcement learning algorithms. Our experimental results, using the DSTC2 dataset and a simulated user model capable of sampling audio turns, reveal that audio embeddings lead to overall higher task success (than without using audio embeddings) with statistically significant results across evaluation metrics and training algorithms.

Audio Embedding-Aware Dialogue Policy Learning

“…Last, this study presents several novelties compared to our preliminary work [5]. We provide a larger experimentation and a much deeper analysis of how, when and why speech representations help to learn better dialogue policies.…”

“…User Audio Sampler. Since audio signals need to be fed to the proposed dialogue policies, we employ the User Audio Sampler proposed in [5] to sample an audio turn from the corpus. First, it selects output candidates filtering the audios of dialogue turns labelled with the same dialogue acts and slots generated by the UM.…”

“…3) RL details: REINFORCE and Actor-Critic are policy gradient RL algorithms that learn a set of weights θ in order select action a in state s according to policy π θ (a|s). Our reward function use dense rewards, since previous works suggest that stronger policies are learnt in comparison with sparse rewards [5]. The reward function is as follows: where score is the evaluation score described in section IV-C, which provides intermediate rewards after every system turnespecially if there are any new components in any of the aforementioned task completion metrics.…”

“…available [2]- [4]. Preliminary experiments by [5] performed on the DSTC2 [6] spoken dialogue corpus indicate that, audio embeddings might actually encode useful acoustic information and exploit it to learn better dialogue policies, when combined with GPT-2 transformer [7] based neural dialogue policies.…”

A survey on deep reinforcement learning for audio-based applications

“…DRL agents are often trained from scratch instead of inheriting useful behaviours from other agents. Some agents from Table 4 [such as (Williams and Zweig 2016;Liu et al 2017;Zorrilla et al 2021)] have avoided learning from scratch by showing that applying DRL on top of non-DRL or supervised methods yields improved performance due to the optimisation element that DRL brings instead of only mimicking demonstration data. But those systems typically focus a single dataset and the idea of transferring useful and effective knowledge from other/many tasks to a new or targeted task remains to be demonstrated.…”