Generating Body Motions using Spoken Language in Dialogue

Ishii, Ryo; Katayama, Taichi; Higashinaka, Ryuichiro; Tomita, Junji

doi:10.1145/3267851.3267866

Cited by 24 publications

(32 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 79 ) remains a centrally important empirical challenge. Using increasingly realistic generative models of faces, 43 scenes, 80 bodies, 81 , 82 and voices, 83 plus virtual reality technologies and new statistical tools, 11 , 35 our future work will address these major challenges. Relatedly, our results are based on white Western participants interpreting facial expressions displayed by same-ethnicity faces, using commonly used English-language emotion terms.…”

Section: Resultsmentioning

confidence: 99%

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

et al. 2022

View full text Add to dashboard Cite

Summary Human facial expressions are complex, multi-component signals that can communicate rich information about emotions, 1 , 2 , 3 , 4 , 5 including specific categories, such as “anger,” and broader dimensions, such as “negative valence, high arousal.” 6 , 7 , 8 An enduring question is how this complex signaling is achieved. Communication theory predicts that multi-component signals could transmit each type of emotion information—i.e., specific categories and broader dimensions—via the same or different facial signal components, with implications for elucidating the system and ontology of facial expression communication. 9 We addressed this question using a communication-systems-based method that agnostically generates facial expressions and uses the receiver’s perceptions to model the specific facial signal components that represent emotion category and dimensional information to them. 10 , 11 , 12 First, we derived the facial expressions that elicit the perception of emotion categories (i.e., the six classic emotions 13 plus 19 complex emotions 3 ) and dimensions (i.e., valence and arousal) separately, in 60 individual participants. Comparison of these facial signals showed that they share subsets of components, suggesting that specific latent signals jointly represent—i.e., multiplex—categorical and dimensional information. Further examination revealed these specific latent signals and the joint information they represent. Our results—based on white Western participants, same-ethnicity face stimuli, and commonly used English emotion terms—show that facial expressions can jointly represent specific emotion categories and broad dimensions to perceivers via multiplexed facial signal components. Our results provide insights into the ontology and system of facial expression communication and a new information-theoretic framework that can characterize its complexities.

show abstract

Section: Resultsmentioning

confidence: 99%

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Secondly, text should be taken into account, e.g., as in [17]. Gestures that co-occur with speech depend greatly on the semantic content of the utterance.…”

Section: Future Workmentioning

confidence: 99%

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Kucherenko

Hasegawa

Henter

et al. 2019

Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents

116

107

View full text Add to dashboard Cite

This paper presents a novel framework for automatic speech-driven gesture generation, applicable to human-agent interaction including both virtual agents and robots. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates.Our approach consists of two steps. First, we learn a lowerdimensional representation of human motion using a denoising autoencoder neural network, consisting of a motion encoder Mo-tionE and a motion decoder MotionD. The learned representation preserves the most important aspects of the human pose variation while removing less relevant variation. Second, we train a novel encoder network SpeechE to map from speech to a corresponding motion representation with reduced dimensionality. At test time, the speech encoder and the motion decoder networks are combined: SpeechE predicts motion representations based on a given speech signal and MotionD then decodes these representations to produce motion sequences.We evaluate different representation sizes in order to find the most effective dimensionality for the representation. We also evaluate the effects of using different speech features as input to the model. We find that mel-frequency cepstral coefficients (MFCCs), alone or combined with prosodic features, perform the best. The results of a subsequent user study confirm the benefits of the representation learning.

show abstract

“…These techniques can directly generate human-like nonverbal behaviors of agents designed with skeletal information similar to humans. Moreover, several studies have estimated nonverbal behaviors using spoken languages [7,8,20,22,31,33]. These studies worked on estimating the labels of nonverbal behaviors.…”

Section: Related Work 21 Nonverbal Behavior Generationmentioning

confidence: 99%

“…The prediction of structured output labels has been shown to improve with a conditional random field (CRF) layer on top of the Bi-LSTM [17,26]. We used both personality and text information to estimate the energy function in a CRF layer that predicted the label for each nonverbal behavior type [20] in Table 1. A summary of this model is shown in Figure 2.…”

Section: Generation Modelsmentioning

confidence: 99%

“…For example, nonverbal behaviors accompanying an utterance have the effect of strengthening the persuasive power of that utterance, making it easier for the other party to understand the content of the utterance [27]. Thus, many studies have been conducted to enable virtual agents to automatically generate appropriate nonverbal behaviors [2,4,5,7,8,13,16,20,22,23,25,31,33,39,41].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Impact of Personality on Nonverbal Behavior Generation

Ishii

Ahuja

Nakano

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

Self Cite

View full text Add to dashboard Cite

To realize natural-looking virtual agents, one key technical challenge is to automatically generate nonverbal behaviors from spoken language. Since nonverbal behavior varies depending on personality, it is important to generate these nonverbal behaviors to match the expected personality of a virtual agent. In this work, we study how personality traits relate to the process of generating individual nonverbal behaviors from the whole body, including the head, eye gaze, arms, and posture. To study this, we first created a dialogue corpus including transcripts, a broad range of labelled nonverbal behaviors, and the Big Five personality scores of participants in dyad interactions. We constructed models that can predict each nonverbal behavior label given as an input language representation from the participants' spoken sentences. Our experimental results show that personality can help improve the prediction of nonverbal behaviors.

show abstract

Generating Body Motions using Spoken Language in Dialogue

Cited by 24 publications

References 14 publications

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Impact of Personality on Nonverbal Behavior Generation

Contact Info

Product

Resources

About