Engagement Recognition in Spoken Dialogue via Neural Network by Aggregating Different Annotators' Models

Abstract

This paper addresses engagement recognition based on four multimodal listener behaviors-backchannels, laughing, eyegaze, and head nodding. Engagement is an indicator of how much a user is interested in the current dialogue. Multiple third-party annotators give ground truth labels of engagement in a human-robot interaction corpus. Since perception of engagement is subjective, the annotations are sometimes different between individual annotators. Conventional methods directly use integrated labels, such as those generated through simple majority voting, and do not consider each annotator's recognition. We propose a two-step engagement recognition where each annotator's recognition is modeled and the different annotators' models are aggregated to recognize the integrated label. The proposed neural network consists of two parts. The first part corresponds to each annotator's model which is trained with the corresponding labels independently. The second part aggregates the different annotators' models to obtain one integrated label. After each part is pre-trained, the whole network is fine-tuned through back-propagation of prediction errors. Experimental results show that the proposed network outperforms baseline models which directly recognize the integrated label without considering differing annotations.

Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems

“…Engagement Metrics Engagement is a substantial metric that shows user willingness to continue conversing with the system (Yu, Aoki, and Woodruff 2004;Ma 2018;Inoue et al 2018) and has been studied in the context of dialogue systems (Yu et al 2016;Zhang et al 2018;See et al 2019). Many researchers have considered engagement as a useful metric toward achieving better dialogue systems (Yu et al 2016;Zhang et al 2018).…”

Distinguishing Engagement Facets: An Essential Component for AI-based Healthcare

“…Salam et al [43] used person to detect both individual and group engagement. [26,27] combined different aspects like backchannels, eye gaze, head nodding-based features to detect engagement level. Ben et al [4] combined several attributes like speech and facial expressions, gaze and head motion, distance to robot to identify disengagement.…”

Mitigating Boredom Using An Empathetic Conversational Agent

“…While most of the work has been done to identify basic prototypic emotions, there have been some work in building automated systems to detect more complex emotional and cognitive states. For example, researchers in the education domain have attempted to build automatic models to detect student engagement, a phenomenon inversely related to boredom, from face [6,45] and speech [21]. Researchers have also attempted to model mind-wandering, another phenomenon related to boredom, using eye-gaze behavior [20,29] and physiological signals [5].…”