A conversation robot using head gesture recognition as para-linguistic information

Fujie, Shinya; Ejiri, Y.; Nakajima, Kanako; Matsusaka, Yosuke; Kobayashi, Tetsuo

doi:10.1109/roman.2004.1374748

Cited by 45 publications

(22 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fujie et al [11] used Hidden Markov Models to perform head nod recognition. In their paper, they combined head gesture detection with prosodic low-level features from the same person to determine strongly positive, weak positive and negative responses to yes/no type utterances.…”

Section: Previous Workmentioning

confidence: 99%

A probabilistic multimodal approach for predicting listener backchannels

Morency

Kok

Gratch

2009

Auton Agent Multi-Agent Syst

121

View full text Add to dashboard Cite

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.Keywords Listener backchannel feedback · Nonverbal behavior prediction · Sequential probabilistic model · Conditional random field · Head nod · Multimodal

show abstract

Section: Previous Workmentioning

confidence: 99%

A probabilistic multimodal approach for predicting listener backchannels

Morency

Kok

Gratch

2009

Auton Agent Multi-Agent Syst

121

View full text Add to dashboard Cite

show abstract

“…For example, [16] and [17] test the use of machine learning algorithms in order to recognise some of the functions of head movements in annotated corpora from various languages. [18], [19], [20], [21] predict the feedback head nods and shakes in interactions between humans or between humans and robots using multimodal information such as speech and gaze.…”

Section: Background Workmentioning

confidence: 99%

Transfer learning in multimodal corpora

Navarretta

2013

2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom)

View full text Add to dashboard Cite

People use both their speech and their body when they communicate face to face, thus human communication is multimodal. The development of multimodal coginfocom systems requires models of the relation between the various modalities, but many studies have shown that multimodal behaviours depend on numerous factors comprising the culture, the setting and the communicative situation. Thus, annotated multimodal corpora of different types must be produced. However, annotating multimodal corpora is extremely resource consuming. Therefore, it is important to reuse existing resources also for annotating unseen data in different domains.The main aims of this paper are to investigate a) the distance between the annotations of two multimodal corpora of different type, the extent to which the annotations of a corpus can be used as training data to identify communicative behaviours in the second corpus automatically, c) the effect of the amount of annotations on classification. The results of our study indicate that using the annotations of one corpus to annotate specific communicative phenomena in another corpus gives good results with respect to a simple majority classifier, but they also confirm that multimodal behaviours vary extensively from one type of conversation to the other. Our experiments also indicate that the results of supervised learning on conversational data of limited size can be improved by using the annotations of corpora of different types.

show abstract

“…For such estimation, an image processing technique is useful. Based on a measurement using image processing, methods have been proposed for ND from the motion of the face using an optical flow (Chu and Tanaka, 2012) (Fujie, et al, 2004). However, this method is influenced by background noise.…”

Section: Introductionmentioning

confidence: 99%

A nodding detection system based on the active appearance model

Jindai

Nakamura

Watanabe

2016

JAMDSM

View full text Add to dashboard Cite

In face-to-face conversation, embodied rhythms between speech and body movements such as nodding are mutually synchronized between talkers such as a speaker and a listener. This synchrony is called entrainment in communication and it generates the sharing of embodiment in human interaction. Embodied communication, which is closely related to behavioral and physiological entrainment, is an essential form of communication that forms the basis of interaction between talkers through mutual embodiment. In particular, nodding has an important role as a regulator in embodied interaction and communication. The detection of nodding is useful for an estimation of the activity of conversation. In this paper, we develop a nodding detection system using a nodding detection model based on an analysis of the head movement involved in nodding. This model detects nodding based on the rotational movement of the head, which is estimated from the face tracking of the Active Appearance Model. The effectiveness of the system is demonstrated by comparing the result of the nodding detection results with visual inspection. The developed nodding detection system is demonstrated in an actual event for children.

show abstract

A conversation robot using head gesture recognition as para-linguistic information

Cited by 45 publications

References 4 publications

A probabilistic multimodal approach for predicting listener backchannels

A probabilistic multimodal approach for predicting listener backchannels

Transfer learning in multimodal corpora

A nodding detection system based on the active appearance model

Contact Info

Product

Resources

About