Impact of Personality on Nonverbal Behavior Generation

Ishii, Ryo; Ahuja, Chaitanya; Nakano, Yukiko I.; Morency, Louis–Philippe

doi:10.1145/3383652.3423908

Cited by 14 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Some interesting developments we observe in the recent work are the shift toward neural network [21,30,19,16,1,23,39,31], the use of adversarial learning [19,16], and the use of word embedding [1,23,31]. Neural network has been successful in recent years, which therefore makes it into a reasonable choice for machine learning problems.…”

Section: Related Workmentioning

confidence: 99%

Sequence-to-Sequence Predictive Model: From Prosody to Communicative Gestures

Yunus

Clavel

Pélachaud

2021

Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior

View full text Add to dashboard Cite

Communicative gestures and speech acoustic are tightly linked. Our objective is to predict the timing of gestures according to the acoustic. That is, we want to predict when a certain gesture occurs. We develop a model based on a recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech acoustic and the gesture phases and types have been annotated. The input of the model is a sequence of speech acoustic and the output is a sequence of gesture classes. The classes we are using for the model output is based on a combination of gesture phases and gesture types. We use a sequence comparison technique to evaluate the model performance. We find that the model can predict better certain gesture classes than others. We also perform ablation studies which reveal that fundamental frequency is a relevant feature for gesture prediction task. In another sub-experiment, we find that including eyebrow movements as acting as beat gesture improves the performance. Besides, we also find that a model trained on the data of one given speaker also works for the other speaker of the same conversation. We also perform a subjective experiment to measure how respondents judge the naturalness, the time consistency, and the semantic consistency of the generated gesture timing of a virtual agent. Our respondents rate the output of our model favorably.

show abstract

Section: Related Workmentioning

confidence: 99%

Sequence-to-Sequence Predictive Model: From Prosody to Communicative Gestures

Yunus

Clavel

Pélachaud

2021

Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior

View full text Add to dashboard Cite

show abstract

“…Prior studies have shown that human-centered traits are characterized by specific non-verbal (and specifically, head-motion) behaviors; e.g., an upright posture maintaining eye-contact conveys high Conscientiousness, while gaze avoidance is indicative of low Conscientiousness [16,18]. Likewise, frequent head nodding is seen as courteous and agreeable behavior, while head shaking and frowning indicates a cold demeanor [17,26]. Kinemes inherently enable discovery of temporal head-motion patterns characteristic of a given trait, and sequence learning methods such as Hidden Markov Models (HMMs) and Long-short term memory (LSTM) networks can be employed to learn these latent temporal signatures for continuous or categorical trait prediction.…”

Section: Trait Prediction From Kinemesmentioning

confidence: 99%

“…The head motion time-series is then unsupervisedly decomposed into a kineme sequence comprising elementary head motion units. This decomposition enables discovery of kinemes characteristic of a target trait, e.g., head nodding is typical of courteous or sympathetic behavior [17,26]. In addition to head motion, we also utilize action units (AUs) which encode facial motion and expressions, for trait prediction.…”

mentioning

confidence: 99%

Head Matters: Explainable Human-centered Trait Prediction from Head Motion Dynamics

Madan

Gahalawat

Guha

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

We demonstrate the utility of elementary head-motion units termed kinemes for behavioral analytics to predict personality and interview traits. Transforming head-motion patterns into a sequence of kinemes facilitates discovery of latent temporal signatures characterizing the targeted traits, thereby enabling both efficient and explainable trait prediction. Utilizing Kinemes and Facial Action Coding System (FACS) features to predict (a) OCEAN personality traits on the First Impressions Candidate Screening videos, and (b) Interview traits on the MIT dataset, we note that: (1) A Long-Short Term Memory (LSTM) network trained with kineme sequences performs better than or similar to a Convolutional Neural Network (CNN) trained with facial images; (2) Accurate predictions and explanations are achieved on combining FACS action units (AUs) with kinemes, and (3) Prediction performance is affected by the time-length over which head and facial movements are observed. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Computer vision representations.

show abstract

“…However, the authors do not provide a detailed quantitative mapping of personality dimensions to non-verbal behaviour. Ishii et al (2020) have taken a machine learningbased approach to investigate the link between personality and non-verbal behaviour. Based on an annotated corpus, they have trained a model to generate non-verbal behaviour associated with speech.…”

Section: Related Workmentioning

confidence: 99%

Expressing Personality Through Non-verbal Behaviour in Real-Time Interaction

2021

View full text Add to dashboard Cite

The attribution of traits plays an important role as a heuristic for how we interact with others. Many psychological models of personality are analytical in that they derive a classification from reported or hypothesised behaviour. In the work presented here, we follow the opposite approach: Our personality model generates behaviour that leads an observer to attribute personality characteristics to the actor. Concretely, the model controls all relevant aspects of non-verbal behaviour such as gaze, facial expression, gesture, and posture. The model, embodied in a virtual human, affords to realistically interact with participants in real-time. Conceptually, our model focuses on the two dimensions of extra/introversion and stability/neuroticism. In the model, personality parameters influence both, the internal affective state as well as the characteristic of the behaviour execution. Importantly, the parameters of the model are based on empirical findings in the behavioural sciences. To evaluate our model, we conducted two types of studies. Firstly, passive experiments where participants rated videos showing variants of behaviour driven by different personality parameter configurations. Secondly, presential experiments where participants interacted with the virtual human, playing rounds of the Rock-Paper-Scissors game. Our results show that the model is effective in conveying the impression of the personality of a virtual character to users. Embodying the model in an artificial social agent capable of real-time interactive behaviour is the only way to move from an analytical to a generative approach to understanding personality, and we believe that this methodology raises a host of novel research questions in the field of personality theory.

show abstract

Impact of Personality on Nonverbal Behavior Generation

Cited by 14 publications

References 30 publications

Sequence-to-Sequence Predictive Model: From Prosody to Communicative Gestures

Sequence-to-Sequence Predictive Model: From Prosody to Communicative Gestures

Head Matters: Explainable Human-centered Trait Prediction from Head Motion Dynamics

Expressing Personality Through Non-verbal Behaviour in Real-Time Interaction

Contact Info

Product

Resources

About