Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

Abstract

Engagement represents how much a user is interested in and willing to continue the current dialogue. Engagement recognition will provide an important clue for dialogue systems to generate adaptive behaviors for the user. This paper addresses engagement recognition based on multimodal listener behaviors of backchannels, laughing, head nodding, and eye gaze. In the annotation of engagement, the ground-truth data often differs from one annotator to another due to the subjectivity of the perception of engagement. To deal with this, we assume that each annotator has a latent character that affects his/her perception of engagement. We propose a hierarchical Bayesian model that estimates both engagement and the character of each annotator as latent variables. Furthermore, we integrate the engagement recognition model with automatic detection of the listener behaviors to realize online engagement recognition. Experimental results show that the proposed model improves recognition accuracy compared with other methods which do not consider the character such as majority voting. We also achieve online engagement recognition without degrading accuracy.

Semi-autonomous avatar enabling unconstrained parallel conversations –seamless hybrid of WOZ and autonomous dialogue systems–

“…are less likely to keep the user engaged in the conversation. Besides these linguistic features, we are considering use of non-linguistic features such as backchannels, laughing, head nodding, and eye-gaze [14]. We trained the engagement recognition model using only linguistic features and confirmed that the recognition accuracy was 70.0%.…”

Embodiment in Dialogue: Daily Dialogue Android Based on Multimodal Information

“…For estimating preferences, Kobayashi et al [38] developed a dialogue system that estimates user's preferences in dialogues by recognizing the polarity of user utterances using syntax, surface case, and deep case analysis. Other methods estimate user interests by combining linguistic and non-linguistic information [39,40]. Although estimating user preferences for the current item or topic is possible, no relation between the preferences for other items, that is, the concept of user's preferences, is estimated.…”

Engagement in Human-Agent Interaction: An Overview

“…Papers can be divided into those that consider engagement as a process and those that treat engagement as a state. The state point of view assumes that one is either engaged or not engaged (e.g., Inoue et al, 2018 ), while the process point of view assumes that there are different processes that unfold during an interaction. Here the action of getting engaged is part of the construct of engagement itself.…”