Latent Character Model for Engagement Recognition Based on Multimodal Behaviors

Inoue, Koji; Lala, Divesh; Takanashi, Katsuya; Kawahara, Tatsuya

doi:10.1007/978-981-13-9443-0_11

Cited by 15 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For estimating preferences, Kobayashi et al [13] developed a dialogue system that estimates user's preferences in dialogues by recognizing the polarity of user utterances using syntax, surface case, and deep case analysis. Other methods estimate user interests by combining linguistic and non-linguistic information [9,32]. Although estimating user preferences for the current item or topic is possible, no relation between the preferences for other items, that is, the concept of user's preferences, is estimated.…”

Section: Related Workmentioning

confidence: 99%

Female-Type Android’s Drive to Quickly Understand a User’s Concept of Preferences Stimulates Dialogue Satisfaction: Dialogue Strategies for Modeling User’s Concept of Preferences

Uchida

Minato

Nakamura

et al. 2021

Int J of Soc Robotics

View full text Add to dashboard Cite

This research develops a conversational robot that stimulates users’ dialogue satisfaction and motivation in non-task-oriented dialogues that include opinion and/or preference exchanges. One way to improve user satisfaction and motivation is by demonstrating the robot’s ability to understand user opinions. In this paper, we explore a method that efficiently obtains the concept of user preferences: likes and dislikes. The concept is acquired by complementing a small amount of user preference data observed in dialogues. As a method for efficient collection, we propose a dialogue strategy that creates utterances with the largest expected complementation. Our experimental results with a female-type android robot suggest that the proposed strategy efficiently obtained user preferences and enhanced dialogue satisfaction. In addition, the strength of user motivation (i.e., long-term willingness to communicate with the android) is only positively correlated with the android’s willingness to understand. Our results not only show the effectiveness of our proposed strategy but also suggest a design theory for dialogue robots to stimulate dialogue motivation, although the current results are derived only from a female-type android.

show abstract

Section: Related Workmentioning

confidence: 99%

Female-Type Android’s Drive to Quickly Understand a User’s Concept of Preferences Stimulates Dialogue Satisfaction: Dialogue Strategies for Modeling User’s Concept of Preferences

Uchida

Minato

Nakamura

et al. 2021

Int J of Soc Robotics

View full text Add to dashboard Cite

show abstract

“…One approach involves focusing on how to define reliable labels from subjectively annotated labels. The other approach explicitly trains the differences in the labels produced by multiple coders [17,23,29].Earlier studies explored approaches for avoiding coder disagreement and merging different labels given by the multiple coders recruited through crowdsourcing via majority voting [36]. Ozkan et al [29] proposed a two-step conditional random field (CRF), which was designed for a backchannel prediction task.…”

Section: With Subjectively Annotated Labelsmentioning

confidence: 99%

“…Ozkan et al [29] proposed a two-step conditional random field (CRF), which was designed for a backchannel prediction task. Inoue et al [17] proposed a recognition model for user engagement in human-robot interactions using a hierarchical Bayesian model that estimates both the engagement level and the characteristics of each coder as latent variables. Kumano et al [23] proposed a probabilistic model for integrating labels of empathy as perceived by coders.…”

Section: With Subjectively Annotated Labelsmentioning

confidence: 99%

Recognizing Social Signals with Weakly Supervised Multitask Learning for Multimodal Dialogue Systems

Hirano

Okada

Komatani

2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Social signal processing is a methodology that is used to infer human inner states, including attitudes, sentiments and impressions, from verbal and nonverbal multimodal information. The difficulty in training a social signal recognition model is that the ground-truth (target) labels given by multiple coders often disagree because the annotation of social signals such as sentiments is a subjective and ambiguous task. We introduce weakly supervised learning (WSL) algorithms to such an inaccurate supervision setting in which the target label is not necessarily accurate. The novel challenge in this paper is to explore an effective WSL strategy for recognizing social signals. The strategy is verified through two multimodal datasets including audio, visual, and linguistic data collected in a human-agent dialogue setting. First, we clarify that the proposed WSL strategy for deep neural networks (DNNs), called tri-teaching works well in almost all classification tasks. Second, we demonstrate the effectiveness of integrating WSL and multitask learning (MTL), which exploits several label types in the datasets. Third, we show that our proposed approach achieves less accuracy degradation than an existing training algorithm for a DNN (curriculum learning) in a cross-corpus setting, with a maximum improvement of 7.2%. CCS CONCEPTS• Computing methodologies → Discourse, dialogue and pragmatics; Multitask learning.

show abstract

“…To implement an adaptive dialogue system, it is important to recognize the user's engagement, interest, and sentiment (e.g., enjoyment during the conversation) based on multimodal behaviors, and many studies have focused on these factors [2,11,23]. In [11], a recognition model for user engagement (interest and willingness to continue the dialogue) in human-robot interactions was proposed based on the user's audio-visual information. In [35], to assess the presence of the interest of a user in a time series, they considered an exchange between the system and user as a unit in a chat dialogue.…”

Section: Related Workmentioning

confidence: 99%

Is She Truly Enjoying the Conversation?

Katada

Okada

Hirano

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

In human-agent interactions, it is necessary for the systems to identify the current emotional state of the user to adapt their dialogue strategies. Nevertheless, this task is challenging because the current emotional states are not always expressed in a natural setting and change dynamically. Recent accumulated evidence has indicated the usefulness of physiological modalities to realize emotion recognition. However, the contribution of the time series physiological signals in human-agent interaction during a dialogue has not been extensively investigated. This paper presents a machine learning model based on physiological signals to estimate a user's sentiment at every exchange during a dialogue. Using a wearable sensing device, the time series physiological data including the electrodermal activity (EDA) and heart rate in addition to acoustic and visual information during a dialogue were collected. The sentiment labels were annotated by the participants themselves and by external human coders for each exchange consisting of a pair of system and participant utterances. The experimental results showed that a multimodal deep neural network (DNN) model combined with the EDA and visual features achieved an accuracy of 63.2%. In general, this task is challenging, as indicated by the accuracy of 63.0% attained by the external coders. The analysis of the sentiment estimation results for each individual indicated that the human coders often wrongly estimated the negative sentiment labels, and in this case, the performance of the DNN model was higher than that of the human coders. These results indicate that physiological signals can help in detecting the implicit aspects of negative sentiments, which are acoustically/visually indistinguishable. CCS CONCEPTS • Computing methodologies → Discourse, dialogue and pragmatics; Physiological model.

show abstract

Latent Character Model for Engagement Recognition Based on Multimodal Behaviors

Cited by 15 publications

References 23 publications

Female-Type Android’s Drive to Quickly Understand a User’s Concept of Preferences Stimulates Dialogue Satisfaction: Dialogue Strategies for Modeling User’s Concept of Preferences

Female-Type Android’s Drive to Quickly Understand a User’s Concept of Preferences Stimulates Dialogue Satisfaction: Dialogue Strategies for Modeling User’s Concept of Preferences

Recognizing Social Signals with Weakly Supervised Multitask Learning for Multimodal Dialogue Systems

Is She Truly Enjoying the Conversation?

Contact Info

Product

Resources

About