Facial Gestures: Taxonomy and Application of Nonverbal, Nonemotional Facial Displays for Embodied Conversational Agents

Abstract

Facial displays are an extremely important communication channel fulfilling a wide variety of functions in discourse and conversation. Humans use them naturally, often subconsciously, and are therefore very sensitive to the application of such displays in their computer-generated correspondents, Embodied Conversational Agents (ECA). In this chapter, we aim to provide an extensive survey of one class of facial displays, the facial gestures. Facial gestures include various nods and head movements, blinks, eyebrow gestures and gaze, i.e., all facial displays except explicit verbal and emotional displays (visemes or expressions such as smile). Consciously or subconsciously, facial gestures play an important role both in discourse and in conversation. They are instrumental in turn taking, emphasizing, providing rhythm, and can be connected to physiological functions. While verbal and emotional displays may be regarded as the explicit, perhaps even obvious, the facial gestures are less tangible -yet they are largely responsible for what we intuitively call natural behavior of the face. In other words, an ECA pronouncing a sentence using perfect coarticulation mechanism for the lips and displaying a carefully modeled expression of surprise will still look unnatural if the facial gestures are not right as well. It is therefore extremely important for an ECA to implement facial gestures well. While there is a large body of knowledge on this topic both from psychology and ECA literature, it is quite scattered. Existing ECA implementations typically concentrate on some aspects of facial gestures but do not cover the complete set. We attempt to provide a complete survey of facial gestures that can be useful as a guideline for their implementation in an ECA. Specifically, we provide a systematically organized repertoire of usual facial gestures. For each gesture class, we provide the information on its typical usage in discourse and conversation, conscious or subconscious causes, any available knowledge on Conversational Informatics: An Engineering Approach Edited by Toyoaki Nishida © 2007 John Wiley & Sons, Ltd typical dynamics and amplitude of the gesture. Finally we provide an example of a practical system implementation as a case study.

Semantic Crime Investigation System

“…The example in the case study in this article focus on the eye signals, eyes are a very important part of the non-verbal signals we send to others. Kinesics ontology contains 15 classes and 319 instances [14]- [16]. Sample of it was illustrated in Fig.…”

Untitled

“…Verbal communication is a way of conveying one's emotion or meaning via words while non-verbal communication means exchanging information via gestures, facial expressions, or voice rather than words (Knapp & Hall, 2002). Simply put, non-verbal communication is considered an important means of delivering information to others without words (Burgoon, Birk, & Pfau, 1990;Zoric, Smid, & Pandzic, 2007). Birdwhistel (1970) found that words account for just a third of communication, and the remaining two thirds consist of non-verbal channels.…”

“…Paralanguage is relevant to the vocal features of language, such as tone, pace of speech, or pitch (Sikorski, 2012); even laughter or no fluencies (e.g., um, ah) are part of paralanguage (Jacobi, 2009). Physical appearance refers to the outward look of a person and includes physical attractiveness, clothing, or hairstyle (Zoric et al, 2007). Proxemics refers to the amount of space and proximity that people feel it necessary to set between themselves and others to have a conversation (Harrigan, 2009).…”

Understanding Your Students' Nonverbal Communication: A Primer for the New Engineering Educator

“…While Miller (2005a) states that nonverbal communication includes facial expressions, eye contact, touching, tone of voice, dress, posture, and spatial distance. Perhaps one of the most comprehensive and scientific categorizations of nonverbal communication is offered by Zoric, Smid et al (2007).…”

“…While it is useful to consider and discuss each category of nonverbal communication individually, it is important to realize that they rarely occur alone. Zoric, Smid et al (2007) describe nonverbal conditions occurring in clusters (multiple displays at one time).…”