A Methodology for the Automatic Extraction and Generation of Non-Verbal Signals Sequences Conveying Interpersonal Attitudes

Abstract

Depending on their application, Embodied Conversational Agents (ECAs) must be able to express various affects or social constructs such as emotions or social attitudes. Non-verbal signals, such as smiles or gestures, contribute to the expression of attitudes. Social attitudes affect the whole behavior of a person: as Scherer puts it, they are "characteristic of an affective style that colors the entire interaction" [1]. Moreover, recent findings have demonstrated that non-verbal signals are not interpretated in isolation but along with surrounding signals: for instance, a smile followed by a gaze aversion and a head aversion may signal embarassment rather than amusement [2]. Non-verbal behavior planning models designed to allow ECAs to express attitudes should thus consider complete sequences of non-verbal signals and not only signals independently of one another. However, existing models do not take this into account, or in a limited manner. The contribution of this paper is a methodology for the automatic extraction of sequences of non-verbal signals characteristic of a social phenomenon from a multimodal corpus, and a non-verbal behavior planning model that takes into account sequences of non-verbal signals rather than signals independently. This methodology is applied to design a virtual recruiter capable of expressing social attitudes, which is then evaluated in and out of an interaction context.

Leveraging the Dynamics of Non-Verbal Behaviors For Social Attitude Modeling

“…Attitudes are expressed through both verbal and non-verbal behaviors. Several works showed that combining both non-verbal and verbal modalities may lead to better attitude recognition [26], [24]. Our model has only focused on the non-verbal behavior for attitude expression.…”

“…A pattern is a sub-sequence that occurs frequently in the dataset. Sequence mining has been applied in a wide range of reallife applications in many domains such as marketing [35], bioinformatics [49], text mining [50] and human behavior analysis [24], [44]. In this Section, we present the different algorithms of sequence mining.…”

“…Designing virtual agents have gained a lot of attention in the recent years where virtual agents have become more and more present in our everyday lives. They can be used for a variety of applications ranging from education [23] and training [24] to therapy [25]. We aim to endow an ECA with the ability to express different interpersonal attitudes depending on the interaction context.…”

A Corpus‐Based and Complex Computing Digital Media System for 3D Animation

“…Wireless Communications and Mobile Computing based on visual recognition, which uses the motion and expression capture technology in visual recognition to render the animation task action expressions, which improves the efficiency and quality of 3D animation production [13]. Most design companies currently use three main types of 3D animation material management platforms, stand-alone management mode, departmental LAN management mode, and company LAN management mode.…”

Extracting Interpersonal Stance from Vocal Signals

“…As a consequence, a number of authors have developed systems to automatically recognize the interpersonal stance (often operationalised via the Interpersonal Circumplex) displayed by the user based on social signals. One of the most notable frameworks is presented by Chollet et al [5], who defined a multilayer framework that combines various multi-modal signals: gaze, head orientation, facial expression, dialogue sequences and a computational model to infer interpersonal stance. Although the system covers many modalities, voice is only addressed in a limited sense (i.e., the system uses the moment on which the user is speaking, rather than specific features of the voice).…”