Bidirectional LSTM Networks Employing Stacked Bottleneck Features for Expressive Speech-Driven Head Motion Synthesis

Haag, Kathrin; Shimodaira, Hiroshi

doi:10.1007/978-3-319-47665-0_18

Cited by 26 publications

(33 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MLP models are however limited in modelling temporal data. Ding et al [15] and Haag and Shimodaira [14] thus compared the MLP with a bidirectional long short-term memory (BLSTM) model in the head motion synthesis task. Both works reported improvement of the BLSTM-based system over the MLPbased one in terms of the naturalness of the synthesised motion assessed via a user study, root mean-squared error and canonical correlations between the original and synthesised head motion.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, humans tend to feel unsettled when humanoid robots do not act realistically [11], and when there is an incompatibility between speech and associated gestures [12]. Several works developed audio-driven motion synthesis systems targeted at animated conversational agents and talking avatars, for example, for the synthesis of head motion [13][14][15] or hand movements [16,17]. To the best of our knowledge, there has been no previous attempt at whole upper-body (head, hands and torso) motion synthesis from audio on a humanoid robot.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Audio-Driven Robot Upper-Body Motion Synthesis

Ondras

Çeliktutan

Bremner

et al. 2021

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Body language is an important aspect of human communication, which an effective human-robot interaction interface should mimic well. Human beings exchange information and convey their thoughts and feelings through gaze, facial expressions, body language and tone of voice along with spoken words, and infer 65% of the meaning of the communicated messages from these nonverbal cues. Modern robotic platforms are however limited in their ability to automatically generate behaviours that align with their speech. In this paper, we develop a neural network based system that takes audio from a user as an input and generates upper-body gestures including head, hand and torso movements of the user on a humanoid robot, namely, Softbank Robotics' Pepper. Our system was evaluated quantitatively as well as qualitatively using web-surveys when driven by natural speech and synthetic speech. We compare the impact of generic and personspecific neural network models on the quality of synthesised movements. We further investigate the relationships between quantitative and qualitative evaluations and examine how the speaker's personality traits affect the synthesised movements.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Audio-Driven Robot Upper-Body Motion Synthesis

Ondras

Çeliktutan

Bremner

et al. 2021

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“…Several recent works have applied neural networks in this domain [10,11,29,30,34]. Among the cited works, Haag & Shimodaira [11] use a bottleneck network to learn compact representations, although their bottleneck features subsequently are used to define prediction inputs rather than prediction outputs as in the work we presented. Our proposed method works on a different aspect of non-verbal behavior that co-occurs with speech, namely generating body motion driven by speech.…”

Section: Data-driven Head and Face Movementsmentioning

confidence: 99%

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Kucherenko

Hasegawa

Henter

et al. 2019

Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents

115

View full text Add to dashboard Cite

This paper presents a novel framework for automatic speech-driven gesture generation, applicable to human-agent interaction including both virtual agents and robots. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates.Our approach consists of two steps. First, we learn a lowerdimensional representation of human motion using a denoising autoencoder neural network, consisting of a motion encoder Mo-tionE and a motion decoder MotionD. The learned representation preserves the most important aspects of the human pose variation while removing less relevant variation. Second, we train a novel encoder network SpeechE to map from speech to a corresponding motion representation with reduced dimensionality. At test time, the speech encoder and the motion decoder networks are combined: SpeechE predicts motion representations based on a given speech signal and MotionD then decodes these representations to produce motion sequences.We evaluate different representation sizes in order to find the most effective dimensionality for the representation. We also evaluate the effects of using different speech features as input to the model. We find that mel-frequency cepstral coefficients (MFCCs), alone or combined with prosodic features, perform the best. The results of a subsequent user study confirm the benefits of the representation learning.

show abstract

“…Deep Bi-Directional Long Short Term Memory (BLSTM) models appear in Ding et al [24], where they report improvements over their own earlier work. More recently Haag [25] uses BLSTMs and Bottleneck features [26]. In our own earlier work [4], we use a BLSTM based Conditional Variational Autoencoder (CVAE) to model the many-to-many mapping of speech to head pose prediction, both for speaker, and for the head pose of the listener in dyadic conversation [27].…”

Section: Head Posementioning

confidence: 99%

Joint Learning of Facial Expression and Head Pose from Speech

Greenwood¹,

Matthews²,

Laycock³

2018

Interspeech 2018

View full text Add to dashboard Cite

Natural movement plays a significant role in realistic speech animation, and numerous studies have demonstrated the contribution visual cues make to the degree human observers find an animation acceptable. Natural, expressive, emotive, and prosodic speech exhibits motion patterns that are difficult to predict with considerable variation in visual modalities. Recently, there have been some impressive demonstrations of face animation derived in some way from the speech signal. Each of these methods have taken unique approaches, but none have included rigid head pose in their predicted output. We observe a high degree of correspondence with facial activity and rigid head pose during speech, and exploit this observation to jointly learn full face animation and head pose rotation and translation combined. From our own corpus, we train Deep Bi-Directional LSTMs (BLSTM) capable of learning long-term structure in language to model the relationship that speech has with the complex activity of the face. We define a model architecture to encourage learning of rigid head motion via the latent space of the speaker's facial activity. The result is a model that can predict lip sync and other facial motion along with rigid head motion directly from audible speech.

show abstract

Bidirectional LSTM Networks Employing Stacked Bottleneck Features for Expressive Speech-Driven Head Motion Synthesis

Cited by 26 publications

References 21 publications

Audio-Driven Robot Upper-Body Motion Synthesis

Audio-Driven Robot Upper-Body Motion Synthesis

Analyzing Input and Output Representations for Speech-Driven Gesture Generation

Joint Learning of Facial Expression and Head Pose from Speech

Contact Info

Product

Resources

About