Speech Driven Backchannel Generation Using Deep Q-Network for Enhancing Engagement in Human-Robot Interaction

Abstract

We present a novel method for training a social robot to generate backchannels during human-robot interaction. We address the problem within an off-policy reinforcement learning framework, and show how a robot may learn to produce non-verbal backchannels like laughs, when trained to maximize the engagement and attention of the user. A major contribution of this work is the formulation of the problem as a Markov decision process (MDP) with states defined by the speech activity of the user and rewards generated by quantified engagement levels. The problem that we address falls into the class of applications where unlimited interaction with the environment is not possible (our environment being a human) because it may be time-consuming, costly, impracticable or even dangerous in case a bad policy is executed. Therefore, we introduce deep Q-network (DQN) in a batch reinforcement learning framework, where an optimal policy is learned from a batch data collected using a more controlled policy. We suggest the use of human-to-human dyadic interaction datasets as a batch of trajectories to train an agent for engaging interactions. Our experiments demonstrate the potential of our method to train a robot for engaging behaviors in an offline manner.

Engagement Rewarded Actor-Critic with Conservative Q-Learning for Speech-Driven Laughter Backchannel Generation

“…We formulate the problem of backchannel generation as a MDP, as similar to [9]. We use an off-policy actor-critic method to learn from a human-to-human interaction dataset which we employ to sample a batch of trajectories.…”

“…Table 1 shows the OPE evaluation results obtained over the test data for three off-policy RL methods: neural fitted Q-learning (NFQ) [21], batch-DQN [9] and our method (AC-CQL) as well as for a supervised learning (SL) method [30] and for a mirroring policy (MP) which mimics speaker's laughter in the dataset [31]. Mirroring can be thought of as an effective baseline policy since imitating other's laughter is very commonplace in human to human social interactions.…”

“…Although some previous work on offline RL has shown promising results in terms of learning a more engaging backchannel policy than that followed by humans in a dataset [9], the off-policy methods used for offline RL are known to suffer from distributional shift [14]. The distributional shift problem is particularly apparent in the case of backchannel learning, especially for laughs, since laughs are usually very sparse in human-human communication and datasets are rather biased towards rewarded laughs with almost no demonstration of negative laughs and of what happens for example when backchannels are used in excess.…”

“…The distributional shift problem is particularly apparent in the case of backchannel learning, especially for laughs, since laughs are usually very sparse in human-human communication and datasets are rather biased towards rewarded laughs with almost no demonstration of negative laughs and of what happens for example when backchannels are used in excess. Hence off-policy RL methods are prone to overestimate Q-values, that may result in policies generating backchannels more frequently than a human would do [9]. In this paper we aim to address this distributional shift problem by making use of conservative Q-learning, which is a very recently developed method for offline settings [12].…”

Bottom-up discovery of structure and variation in response tokens (‘backchannels’) across diverse languages

“…Other lines of work have used response tokens as a cue to predict the dynamics of talk and turn-taking [11] or to make inferences about mental and cognitive states [12]. A growing amount of work seeks to model feedback behaviour in human-agent interaction, including by means of response token generation [13,14] and attentive listening systems [15,16]. Despite considerable progress, the place of response tokens in speech technology is by no means settled: they tend to be missed by speech recognizers [17,18,19] and dialog managers have a hard time dealing with them [20], showing that they remain a key issue on which progress towards future generations of voice-interactive technologies and conversational user interfaces depends.…”